Download Learning Python

WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
LearningPython
WOW! eBook
www.wowebook.org
TableofContents
LearningPython
Credits
AbouttheAuthor
Acknowledgements
AbouttheReviewers
www.PacktPub.com
Supportfiles,eBooks,discountoffers,andmore
Whysubscribe?
FreeaccessforPacktaccountholders
Preface
Whatthisbookcovers
Whatyouneedforthisbook
Whothisbookisfor
Conventions
Readerfeedback
Customersupport
Downloadingtheexamplecode
Errata
Piracy
Questions
1.IntroductionandFirstSteps–TakeaDeepBreath
Aproperintroduction
EnterthePython
AboutPython
Portability
Coherence
Developerproductivity
Anextensivelibrary
Softwarequality
WOW! eBook
www.wowebook.org
Softwareintegration
Satisfactionandenjoyment
Whatarethedrawbacks?
WhoisusingPythontoday?
Settinguptheenvironment
Python2versusPython3–thegreatdebate
InstallingPython
SettingupthePythoninterpreter
Aboutvirtualenv
Yourfirstvirtualenvironment
Yourfriend,theconsole
HowyoucanrunaPythonprogram
RunningPythonscripts
RunningthePythoninteractiveshell
RunningPythonasaservice
RunningPythonasaGUIapplication
HowisPythoncodeorganized
Howdoweusemodulesandpackages
Python’sexecutionmodel
Namesandnamespaces
Scopes
Objectandclasses
Guidelinesonhowtowritegoodcode
ThePythonculture
AnoteontheIDEs
Summary
2.Built-inDataTypes
Everythingisanobject
Mutableorimmutable?Thatisthequestion
Numbers
Integers
WOW! eBook
www.wowebook.org
Booleans
Reals
Complexnumbers
Fractionsanddecimals
Immutablesequences
Stringsandbytes
Encodinganddecodingstrings
Indexingandslicingstrings
Tuples
Mutablesequences
Lists
Bytearrays
Settypes
Mappingtypes–dictionaries
Thecollectionsmodule
Namedtuples
Defaultdict
ChainMap
Finalconsiderations
Smallvaluescaching
Howtochoosedatastructures
Aboutindexingandslicing
Aboutthenames
Summary
3.IteratingandMakingDecisions
Conditionalprogramming
Aspecializedelse:elif
Theternaryoperator
Looping
Theforloop
Iteratingoverarange
WOW! eBook
www.wowebook.org
Iteratingoverasequence
Iteratorsanditerables
Iteratingovermultiplesequences
Thewhileloop
Thebreakandcontinuestatements
Aspecialelseclause
Puttingthisalltogether
Example1–aprimegenerator
Example2–applyingdiscounts
Aquickpeekattheitertoolsmodule
Infiniteiterators
Iteratorsterminatingontheshortestinputsequence
Combinatoricgenerators
Summary
4.Functions,theBuildingBlocksofCode
Whyusefunctions?
Reducecodeduplication
Splittingacomplextask
Hideimplementationdetails
Improvereadability
Improvetraceability
Scopesandnameresolution
Theglobalandnonlocalstatements
Inputparameters
Argumentpassing
Assignmenttoargumentnamesdon’taffectthecaller
Changingamutableaffectsthecaller
Howtospecifyinputparameters
Positionalarguments
Keywordargumentsanddefaultvalues
Variablepositionalarguments
WOW! eBook
www.wowebook.org
Variablekeywordarguments
Keyword-onlyarguments
Combininginputparameters
Avoidthetrap!Mutabledefaults
Returnvalues
Returningmultiplevalues
Afewusefultips
Recursivefunctions
Anonymousfunctions
Functionattributes
Built-infunctions
Onefinalexample
Documentingyourcode
Importingobjects
Relativeimports
Summary
5.SavingTimeandMemory
map,zip,andfilter
map
zip
filter
Comprehensions
Nestedcomprehensions
Filteringacomprehension
dictcomprehensions
setcomprehensions
Generators
Generatorfunctions
Goingbeyondnext
Theyieldfromexpression
Generatorexpressions
WOW! eBook
www.wowebook.org
Someperformanceconsiderations
Don’toverdocomprehensionsandgenerators
Namelocalization
Generationbehaviorinbuilt-ins
Onelastexample
Summary
6.AdvancedConcepts–OOP,Decorators,andIterators
Decorators
Adecoratorfactory
Object-orientedprogramming
ThesimplestPythonclass
Classandobjectnamespaces
Attributeshadowing
I,me,andmyself–usingtheselfvariable
Initializinganinstance
OOPisaboutcodereuse
Inheritanceandcomposition
Accessingabaseclass
Multipleinheritance
Methodresolutionorder
Staticandclassmethods
Staticmethods
Classmethods
Privatemethodsandnamemangling
Thepropertydecorator
Operatoroverloading
Polymorphism–abriefoverview
Writingacustomiterator
Summary
7.Testing,Profiling,andDealingwithExceptions
Testingyourapplication
WOW! eBook
www.wowebook.org
Theanatomyofatest
Testingguidelines
Unittesting
Writingaunittest
Mockobjectsandpatching
Assertions
Aclassicunittestexample
Makingatestfail
Interfacetesting
Comparingtestswithandwithoutmocks
Boundariesandgranularity
Amoreinterestingexample
Test-drivendevelopment
Exceptions
ProfilingPython
Whentoprofile?
Summary
8.TheEdges–GUIsandScripts
Firstapproach–scripting
Theimports
Parsingarguments
Thebusinesslogic
Secondapproach–aGUIapplication
Theimports
Thelayoutlogic
Thebusinesslogic
Fetchingthewebpage
Savingtheimages
Alertingtheuser
Howtoimprovetheapplication?
Wheredowegofromhere?
WOW! eBook
www.wowebook.org
Thetkinter.tixmodule
Theturtlemodule
wxPython,PyQt,andPyGTK
Theprincipleofleastastonishment
Threadingconsiderations
Summary
9.DataScience
IPythonandJupyternotebook
Dealingwithdata
Settingupthenotebook
Preparingthedata
Cleaningthedata
CreatingtheDataFrame
Unpackingthecampaignname
Unpackingtheuserdata
Cleaningeverythingup
SavingtheDataFrametoafile
Visualizingtheresults
Wheredowegofromhere?
Summary
10.WebDevelopmentDoneRight
WhatistheWeb?
HowdoestheWebwork?
TheDjangowebframework
Djangodesignphilosophy
Themodellayer
Theviewlayer
Thetemplatelayer
TheDjangoURLdispatcher
Regularexpressions
Aregexwebsite
WOW! eBook
www.wowebook.org
SettingupDjango
Startingtheproject
Creatingusers
AddingtheEntrymodel
Customizingtheadminpanel
Creatingtheform
Writingtheviews
Thehomeview
Theentrylistview
Theformview
TyingupURLsandviews
Writingthetemplates
Thefutureofwebdevelopment
WritingaFlaskview
BuildingaJSONquoteserverinFalcon
Summary
11.DebuggingandTroubleshooting
Debuggingtechniques
Debuggingwithprint
Debuggingwithacustomfunction
Inspectingthetraceback
UsingthePythondebugger
Inspectinglogfiles
Othertechniques
Profiling
Assertions
Wheretofindinformation
Troubleshootingguidelines
Usingconsoleeditors
Wheretoinspect
Usingteststodebug
WOW! eBook
www.wowebook.org
Monitoring
Summary
12.SummingUp–ACompleteExample
Thechallenge
Ourimplementation
ImplementingtheDjangointerface
Thesetup
Themodellayer
Asimpleform
Theviewlayer
Importsandhomeview
Listingallrecords
Creatingrecords
Updatingrecords
Deletingrecords
SettinguptheURLs
Thetemplatelayer
Homeandfootertemplates
Listingallrecords
Creatingandeditingrecords
TalkingtotheAPI
Deletingrecords
ImplementingtheFalconAPI
Themainapplication
Writingthehelpers
Codingthepasswordvalidator
Codingthepasswordgenerator
Writingthehandlers
Codingthepasswordvalidatorhandler
Codingthepasswordgeneratorhandler
RunningtheAPI
WOW! eBook
www.wowebook.org
TestingtheAPI
Testingthehelpers
Testingthehandlers
Wheredoyougofromhere?
Summary
Awordoffarewell
Index
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
LearningPython
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
LearningPython
Copyright©2015PacktPublishing
Allrightsreserved.Nopartofthisbookmaybereproduced,storedinaretrievalsystem,
ortransmittedinanyformorbyanymeans,withoutthepriorwrittenpermissionofthe
publisher,exceptinthecaseofbriefquotationsembeddedincriticalarticlesorreviews.
Everyefforthasbeenmadeinthepreparationofthisbooktoensuretheaccuracyofthe
informationpresented.However,theinformationcontainedinthisbookissoldwithout
warranty,eitherexpressorimplied.Neithertheauthor,norPacktPublishing,andits
dealersanddistributorswillbeheldliableforanydamagescausedorallegedtobecaused
directlyorindirectlybythisbook.
PacktPublishinghasendeavoredtoprovidetrademarkinformationaboutallofthe
companiesandproductsmentionedinthisbookbytheappropriateuseofcapitals.
However,PacktPublishingcannotguaranteetheaccuracyofthisinformation.
Firstpublished:December2015
Productionreference:1171215
PublishedbyPacktPublishingLtd.
LiveryPlace
35LiveryStreet
BirminghamB32PB,UK.
ISBN978-1-78355-171-2
www.packtpub.com
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Credits
Author
FabrizioRomano
Reviewers
SimoneBurol
JulioVicenteTrigoGuijarro
VeitHeller
CommissioningEditor
AkramHussain
AcquisitionEditor
IndrajitDas
ContentDevelopmentEditors
SamanthaGonsalves
AdrianRaposo
TechnicalEditor
SiddhiRane
CopyEditors
JanbalDharmaraj
KevinMcGowan
ProjectCoordinator
KinjalBari
Proofreader
SafisEditing
Indexer
PriyaSane
Graphics
KirkD’Penha
AbhinashSahu
ProductionCoordinator
MelwynD’sa
CoverWork
WOW! eBook
www.wowebook.org
MelwynD’sa
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
AbouttheAuthor
FabrizioRomanowasborninItalyin1975.Heholdsamaster’sdegreeincomputer
scienceengineeringfromtheUniversityofPadova.HeisalsoacertifiedScrummaster.
BeforePython,hehasworkedwithseveralotherlanguages,suchasC/C++,Java,PHP,
andC#.
In2011,hemovedtoLondonandstartedworkingasaPythondeveloperforGlasses
Direct,oneofEurope’sleadingonlineprescriptionglassesretailers.
HethenworkedasaseniorPythondeveloperforTBG(nowSprinklr),oneoftheworld’s
leadingcompaniesinsocialmediaadvertising.AtTBG,heandhisteamcollaboratedwith
FacebookandTwitter.TheywerethefirstintheworldtogetaccesstotheTwitter
advertisingAPI.Hewrotethecodethatpublishedthefirstgeo-narrowcastedpromoted
tweetintheworldusingtheAPI.
HecurrentlyworksasaseniorplatformdeveloperatStudent.com,acompanythatis
revolutionizingthewayinternationalstudentsfindtheirperfecthomeallaroundtheworld
HehasdeliveredtalksonTeachingPythonandTDDwithPythonatthelasttwoeditions
ofEuroPythonandatSkillsmatterinLondon.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Acknowledgements
IwouldliketothankAdrianRaposoandIndrajitDasfromPacktPublishingfortheirhelp
andsupportandgivingmetheopportunitytolivethisadventure.Iwouldalsoliketo
thankeveryoneatPacktPublishingwhohavecontributedtotherealizationofthisbook.
SpecialthanksgotoSiddhiRane,mytechnicaleditor.Thankyouforyourkindness,for
workingveryhard,andforgoingtheextramilejusttomakemehappy.
IwouldliketoexpressmydeepestgratitudetoSimoneBurolandJulioTrigo,whohave
giftedmewithsomeoftheirpreciousfreetime.Theyhavereviewedthebookand
providedmewithinvaluablefeedback.
Abigthankyoutomyteammates,MattBennettandJakubKubaBorys,fortheirinterest
inthisbookandfortheirsupportandfeedbackthatmakesmeabettercodereveryday.
AheartfeltthankyoutoMarco“Tex”Beri,whointroducedmetoPythonwithan
enthusiasmsecondtonone.
AspecialthankstoDr.NaomiCeder,fromwhomIlearnedsomuchoverthelastyear.She
hasgivenmeprecioussuggestionsandhasencouragedmetoembracethisopportunity.
Finally,Iwouldliketothankallmyfriendswhohavesupportedmeinanyway.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
AbouttheReviewers
SimoneBurolisanItaliansoftwaredeveloperwhowasborninTreviso(Italy)in1978.
Heobtainedamaster’sdegreeincomputerscienceengineeringfromtheUniversityof
Padua(Italy),andsincethenworkedinbankingfor5yearsinVenice(Italy).In2010,he
movedtoLondon(UnitedKingdom),whereheworkedinwarehouseautomationfor
OcadoTechnologyandtheninbankingforAlgomi.
JulioVicenteTrigoGuijarroisacomputerscientistandsoftwareengineerwithalmosta
decadeofexperienceinsoftwaredevelopment.HeisalsoacertifiedScrummaster,who
enjoysthebenefitsofusingagilesoftwaredevelopment(ScrumandXP).
Hecompletedhisstudiesincomputerscienceandsoftwareengineeringfromthe
UniversityofAlicante,Spain,in2007.Sincethen,hehasworkedwithseveral
technologiesandlanguages,includingMicrosoftDynamicsNAV,Java,JavaScript,and
Python.
SomeoftheapplicationscoveredbyJulioduringhiscareerincludeRESTfulAPIs,ERPs,
billingplatforms,paymentgateways,ande-commercewebsites.
HehasbeenusingPythononbothpersonalandprofessionalprojectssince2012,andheis
passionateaboutsoftwaredesign,softwarequality,andcodingstandards.
Iwouldliketothankmyparentsfortheirlove,goodadvice,andcontinuoussupport.
IwouldalsoliketothankallmyfriendsthatImetalongtheway,whoenrichedmylife,
formotivatingmeandhelpingmeprogress.
VeitHellerisafullstackdeveloper,mostlyworkingonthebackendsideofwebprojects.
HecurrentlyresidesinBerlinandworksforaprototypicalPythonistacompanynamed
Bright.Inhisfreetime,hewritesinterpretersforvariousprogramminglanguages.
IwouldliketothankthepeopleatBrightforbeingawelcomingcompanythatsupports
meinallmyendeavors,myfriendsandmyfamilyforcopingwithmystrangeness,and
manufacturersofcaffeinateddrinksworldwide.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
www.PacktPub.com
WOW! eBook
www.wowebook.org
Supportfiles,eBooks,discountoffers,and
more
Forsupportfilesanddownloadsrelatedtoyourbook,pleasevisitwww.PacktPub.com.
DidyouknowthatPacktofferseBookversionsofeverybookpublished,withPDFand
ePubfilesavailable?YoucanupgradetotheeBookversionatwww.PacktPub.comandas
aprintbookcustomer,youareentitledtoadiscountontheeBookcopy.Getintouchwith
usat<[email protected]>formoredetails.
Atwww.PacktPub.com,youcanalsoreadacollectionoffreetechnicalarticles,signup
forarangeoffreenewslettersandreceiveexclusivediscountsandoffersonPacktbooks
andeBooks.
https://www2.packtpub.com/books/subscription/packtlib
DoyouneedinstantsolutionstoyourITquestions?PacktLibisPackt’sonlinedigital
booklibrary.Here,youcansearch,access,andreadPackt’sentirelibraryofbooks.
WOW! eBook
www.wowebook.org
Whysubscribe?
FullysearchableacrosseverybookpublishedbyPackt
Copyandpaste,print,andbookmarkcontent
Ondemandandaccessibleviaawebbrowser
WOW! eBook
www.wowebook.org
FreeaccessforPacktaccountholders
IfyouhaveanaccountwithPacktatwww.PacktPub.com,youcanusethistoaccess
PacktLibtodayandview9entirelyfreebooks.Simplyuseyourlogincredentialsfor
immediateaccess.
ToAlanTuring,thefatherofComputerScience.
ToGuidoVanRossum,thefatherofPython.
ToAdrianoRomano,myfather,mybiggestfan.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Preface
ShortlyafterIstartedwriting,afriendaskedmeiftherereallywasaneedofanother
LearningPythonbook.
Anexcellentquestionthatwecouldalsoexpressinanotherform:Whathasthisbookto
offer?WhatmakesthisbookdifferentfromtheaverageintroductorybookonPython?
Ithinktherearetwomaindifferencesandmanygoodreasonswhyyouwouldwanttoread
it.
Firstly,westartwithintroducingsomeimportantprogrammingconcepts.Webuildasolid
foundationbycoveringthecriticalaspectsofthiswonderfullanguage.
Thepacegraduallyincreases,alongwiththedifficultyofthesubjectspresented.Bythe
endofChapter7,Testing,Profiling,andDealingwithExceptions,wewillcoverallthe
fundamentals.
FromChapter8,TheEdges–GUIsandScripts,onward,thebooktakesasteepturn,
whichbringsustodifferencenumbertwo.
Toconsolidatetheknowledgeacquired,thereisnothinglikeworkingonasmallproject.
So,inthesecondpartofthebook,eachchapterdeliversaprojectonadifferentsubject.
Weexplorescripting,graphicalinterfaces,datascience,andwebprogramming.
Eachprojectissmallenoughtofitwithinachapterandyetbigenoughtoberelevant.
Eachchapterisinteresting,conveysamessage,andteachessomethingvaluable.
Afterashortsectionondebugging,thebookendswithacompleteexamplethatwraps
thingsup.Itriedtocraftitsothatyouwillbeabletoexpanditinseveralways.
So,thisisdefinitelynottheusualLearningPythonbook.Itsapproachismuchmore
“hands-on”andpractical.
IwantedtoempoweryoutohelpyoubecomeatruePythonninja.ButIalsodidmybest
toentertainyouandfosteryourlogicalthinkingandcreativityalongtheway.
Now,haveIansweredthequestion?
WOW! eBook
www.wowebook.org
Whatthisbookcovers
Chapter1,IntroductionandFirstSteps–TakeaDeepBreath,introducesyouto
fundamentalprogrammingconcepts.ItguidesyoutogettingPythonupandrunningon
yourcomputerandintroducesyoutosomeofitsconstructs.
Chapter2,Built-inDataTypes,introducesyoutoPythonbuilt-indatatypes.Pythonhasa
veryrichsetofnativedatatypesandthischapterwillgiveyouadescriptionandashort
exampleforeachofthem.
Chapter3,IteratingandMakingDecisions,teachesyouhowtocontroltheflowofyour
codebyinspectingconditions,applyinglogic,andperformingloops.
Chapter4,Functions,theBuildingBlocksofCode,teachesyouhowtowritefunctions.
Functionsarethekeystoreusingcode,toreducingdebuggingtime,andingeneral,to
writingbettercode.
Chapter5,SavingTimeandMemory,introducesyoutothefunctionalaspectsofPython
programming.Thischapterteachesyouhowtowritecomprehensionsandgenerators,
whicharepowerfultoolsthatyoucanusetospeedupyourcodeandsavememory.
Chapter6,AdvancedConcepts–OOP,Decorators,andIterators,teachesyouthebasics
ofobject-orientedprogrammingwithPython.Itshowsyouthekeyconceptsandallthe
potentialsofthisparadigm.Italsoshowsyouoneofthemostbelovedcharacteristicsof
Python:decorators.Finally,italsocoverstheconceptofiterators.
Chapter7,Testing,Profiling,andDealingwithExceptions,teachesyouhowtomakeyour
codemorerobust,fast,andstableusingtechniquessuchastestingandprofiling.Italso
formallydefinestheconceptofexceptions.
Chapter8,TheEdges–GUIsandScripts,guidesyouthroughanexamplefromtwo
differentpointsofview.Theyareattheextremitiesofaspectrum:oneimplementationisa
scriptandtheotheroneapropergraphicaluserinterfaceapplication.
Chapter9,DataScience,introducesafewkeyconceptsandaveryspecialtool,the
JupyterNotebook.
Chapter10,WebDevelopmentDoneRight,introducesthefundamentalsofweb
developmentanddeliversaprojectusingtheDjangowebframework.Theexamplewillbe
basedonregularexpressions.
Chapter11,DebuggingandTroubleshooting,showsyouthemainmethodstodebugyour
codeandsomeexamplesonhowtoapplythem.
Chapter12,SummingUp–ACompleteExample,presentsaDjangowebsitethatactsasan
interfacetoanunderlyingslimAPIwrittenwiththeFalconwebframework.Thischapter
takesalltheconceptscoveredinthebooktothenextlevelandsuggestswheretogotodig
deeperandtakethenextsteps.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Whatyouneedforthisbook
Youareencouragedtofollowtheexamplesinthisbook.Inordertodoso,youwillneeda
computer,anInternetconnection,andabrowser.ThebookiswritteninPython3.4,butit
shouldalsoworkwithanyPython3.*version.Ihavewritteninstructionsonhowtoinstall
Pythononthethreemainoperatingsystemsusedtoday:Windows,Mac,andLinux.Ihave
alsoexplainedhowtoinstallalltheextralibrariesusedinthevariousexamplesand
providedsuggestionsifthereaderfindsanyissuesduringtheinstallationofanyofthem.
Noparticulareditorisrequiredtotypethecode;however,Isuggestthatthosewhoare
interestedinfollowingtheexamplesshouldconsideradoptingapropercoding
environment.Ihavegivensuggestionsonthismatterinthefirstchapter.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Whothisbookisfor
Pythonisthemostpopularintroductoryteachinglanguageinthetopcomputerscience
universitiesintheUS,soifyouarenewtosoftwaredevelopmentorifyouhavelittle
experienceandwouldliketostartoffontherightfoot,thenthislanguageandthisbook
arewhatyouneed.Itsamazingdesignandportabilitywillhelpyoubecomeproductive
regardlessoftheenvironmentyouchoosetoworkwith.
IfyouhavealreadyworkedwithPythonoranyotherlanguage,thisbookcanstillbe
usefultoyoubothasareferencetoPython’sfundamentalsandtoprovideawiderangeof
considerationsandsuggestionscollectedovertwodecadesofexperience.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Conventions
Inthisbook,youwillfindanumberoftextstylesthatdistinguishbetweendifferentkinds
ofinformation.Herearesomeexamplesofthesestylesandanexplanationoftheir
meaning.
Codewordsintext,databasetablenames,foldernames,filenames,fileextensions,
pathnames,dummyURLs,userinput,andTwitterhandlesareshownasfollows:“Open
upaPythonconsole,andtypeimportthis.”
Ablockofcodeissetasfollows:
#wedefineafunction,calledlocal
deflocal():
m=7
print(m)
m=5
print(m)
Whenwewishtodrawyourattentiontoaparticularpartofacodeblock,therelevant
linesoritemsaresetinbold:
#wedefineafunction,calledlocal
deflocal():
m=7
print(m)
m=5
print(m)
Anycommand-lineinputoroutputiswrittenasfollows:
>>>frommathimportfactorial
>>>factorial(5)
120
Newtermsandimportantwordsareshowninbold.Wordsthatyouseeonthescreen,
forexample,inmenusordialogboxes,appearinthetextlikethis:“Toopentheconsole
onWindows,gototheStartmenu,chooseRun,andtypecmd.”
Note
Warningsorimportantnotesappearinaboxlikethis.
Tip
Tipsandtricksappearlikethis.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Readerfeedback
Feedbackfromourreadersisalwayswelcome.Letusknowwhatyouthinkaboutthis
book—whatyoulikedordisliked.Readerfeedbackisimportantforusasithelpsus
developtitlesthatyouwillreallygetthemostoutof.
Tosendusgeneralfeedback,simplye-mail<[email protected]>,andmentionthe
book’stitleinthesubjectofyourmessage.
Ifthereisatopicthatyouhaveexpertiseinandyouareinterestedineitherwritingor
contributingtoabook,seeourauthorguideatwww.packtpub.com/authors.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Customersupport
NowthatyouaretheproudownerofaPacktbook,wehaveanumberofthingstohelp
youtogetthemostfromyourpurchase.
WOW! eBook
www.wowebook.org
Downloadingtheexamplecode
Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.com
forallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbook
elsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilesemaileddirectlytoyou.
WOW! eBook
www.wowebook.org
Errata
Althoughwehavetakeneverycaretoensuretheaccuracyofourcontent,mistakesdo
happen.Ifyoufindamistakeinoneofourbooks—maybeamistakeinthetextorthe
code—wewouldbegratefulifyoucouldreportthistous.Bydoingso,youcansaveother
readersfromfrustrationandhelpusimprovesubsequentversionsofthisbook.Ifyoufind
anyerrata,pleasereportthembyvisitinghttp://www.packtpub.com/submit-errata,
selectingyourbook,clickingontheErrataSubmissionFormlink,andenteringthe
detailsofyourerrata.Onceyourerrataareverified,yoursubmissionwillbeacceptedand
theerratawillbeuploadedtoourwebsiteoraddedtoanylistofexistingerrataunderthe
Erratasectionofthattitle.
Toviewthepreviouslysubmittederrata,goto
https://www.packtpub.com/books/content/supportandenterthenameofthebookinthe
searchfield.TherequiredinformationwillappearundertheErratasection.
WOW! eBook
www.wowebook.org
Piracy
PiracyofcopyrightedmaterialontheInternetisanongoingproblemacrossallmedia.At
Packt,wetaketheprotectionofourcopyrightandlicensesveryseriously.Ifyoucome
acrossanyillegalcopiesofourworksinanyformontheInternet,pleaseprovideuswith
thelocationaddressorwebsitenameimmediatelysothatwecanpursuearemedy.
Pleasecontactusat<[email protected]>withalinktothesuspectedpirated
material.
Weappreciateyourhelpinprotectingourauthorsandourabilitytobringyouvaluable
content.
WOW! eBook
www.wowebook.org
Questions
Ifyouhaveaproblemwithanyaspectofthisbook,youcancontactusat
<[email protected]>,andwewilldoourbesttoaddresstheproblem.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter1.IntroductionandFirstSteps–
TakeaDeepBreath
“Giveamanafishandyoufeedhimforaday.Teachamantofishandyoufeedhimforalifetime.” —Chineseproverb
AccordingtoWikipedia,computerprogrammingis:
“…aprocessthatleadsfromanoriginalformulationofacomputingproblemto
executablecomputerprograms.Programminginvolvesactivitiessuchasanalysis,
developingunderstanding,generatingalgorithms,verificationofrequirementsof
algorithmsincludingtheircorrectnessandresourcesconsumption,and
implementation(commonlyreferredtoascoding)ofalgorithmsinatarget
programminglanguage”.
Inanutshell,codingistellingacomputertodosomethingusingalanguageitunderstands.
Computersareverypowerfultools,butunfortunately,theycan’tthinkforthemselves.So
theyneedtobetoldeverything.Theyneedtobetoldhowtoperformatask,howto
evaluateaconditiontodecidewhichpathtofollow,howtohandledatathatcomesfroma
devicesuchasthenetworkoradisk,andhowtoreactwhensomethingunforeseen
happens,say,somethingisbrokenormissing.
Youcancodeinmanydifferentstylesandlanguages.Isithard?Iwouldsay“yes”and
“no”.It’sabitlikewriting.Everybodycanlearnhowtowrite,andyoucantoo.Butwhat
ifyouwantedtobecomeapoet?Thenwritingaloneisnotenough.Youhavetoacquirea
wholeothersetofskillsandthiswilltakealongerandgreatereffort.
Intheend,itallcomesdowntohowfaryouwanttogodowntheroad.Codingisnotjust
puttingtogethersomeinstructionsthatwork.Itissomuchmore!
Goodcodeisshort,fast,elegant,easytoreadandunderstand,simple,easytomodifyand
extend,easytoscaleandrefactor,andeasytotest.Ittakestimetobeabletowritecode
thathasallthesequalitiesatthesametime,butthegoodnewsisthatyou’retakingthe
firststeptowardsitatthisverymomentbyreadingthisbook.AndIhavenodoubtyou
candoit.Anyonecan,infact,weallprogramallthetime,onlywearen’tawareofit.
Wouldyoulikeanexample?
Sayyouwanttomakeinstantcoffee.Youhavetogetamug,theinstantcoffeejar,a
teaspoon,water,andthekettle.Evenifyou’renotawareofit,you’reevaluatingalotof
data.You’remakingsurethatthereiswaterinthekettleaswellasthekettleisplugged-in,
thatthemugisclean,andthatthereisenoughcoffeeinthejar.Then,youboilthewater
andmaybeinthemeantimeyouputsomecoffeeinthemug.Whenthewaterisready,you
pouritintothecup,andstir.
So,howisthisprogramming?
WOW! eBook
www.wowebook.org
Well,wegatheredresources(thekettle,coffee,water,teaspoon,andmug)andweverified
someconditionsonthem(kettleisplugged-in,mugisclean,thereisenoughcoffee).Then
westartedtwoactions(boilingthewaterandputtingcoffeeinthemug),andwhenbothof
themwerecompleted,wefinallyendedtheprocedurebypouringwaterinthemugand
stirring.
Canyouseeit?Ihavejustdescribedthehigh-levelfunctionalityofacoffeeprogram.It
wasn’tthathardbecausethisiswhatthebraindoesalldaylong:evaluateconditions,
decidetotakeactions,carryouttasks,repeatsomeofthem,andstopatsomepoint.Clean
objects,putthemback,andsoon.
Allyouneednowistolearnhowtodeconstructallthoseactionsyoudoautomaticallyin
reallifesothatacomputercanactuallymakesomesenseofthem.Andyouneedtolearna
languageaswell,toinstructit.
Sothisiswhatthisbookisfor.I’lltellyouhowtodoitandI’lltrytodothatbymeansof
manysimplebutfocusedexamples(myfavoritekind).
WOW! eBook
www.wowebook.org
Aproperintroduction
IlovetomakereferencestotherealworldwhenIteachcoding;Ibelievetheyhelppeople
retaintheconceptsbetter.However,nowisthetimetobeabitmorerigorousandseewhat
codingisfromamoretechnicalperspective.
Whenwewritecode,we’reinstructingacomputeronwhatarethethingsithastodo.
Wheredoestheactionhappen?Inmanyplaces:thecomputermemory,harddrives,
networkcables,CPU,andsoon.It’sawhole“world”,whichmostofthetimeisthe
representationofasubsetoftherealworld.
Ifyouwriteapieceofsoftwarethatallowspeopletobuyclothesonline,youwillhaveto
representrealpeople,realclothes,realbrands,sizes,andsoonandsoforth,withinthe
boundariesofaprogram.
Inordertodoso,youwillneedtocreateandhandleobjectsintheprogramyou’rewriting.
Apersoncanbeanobject.Acarisanobject.Apairofsocksisanobject.Luckily,Python
understandsobjectsverywell.
Thetwomainfeaturesanyobjecthasarepropertiesandmethods.Let’stakeaperson
objectasanexample.Typicallyinacomputerprogram,you’llrepresentpeopleas
customersoremployees.Thepropertiesthatyoustoreagainstthemarethingslikethe
name,theSSN,theage,iftheyhaveadrivinglicense,theire-mail,gender,andsoon.Ina
computerprogram,youstoreallthedatayouneedinordertouseanobjectforthepurpose
you’reserving.Ifyouarecodingawebsitetosellclothes,youprobablywanttostorethe
heightandweightaswellasothermeasuresofyourcustomerssothatyoucansuggestthe
appropriateclothesforthem.So,propertiesarecharacteristicsofanobject.Weusethem
allthetime:“Couldyoupassmethatpen?”–“Whichone?”–“Theblackone.”Here,we
usedthe“black”propertyofapentoidentifyit(mostlikelyamongstablueandared
one).
Methodsarethingsthatanobjectcando.Asaperson,Ihavemethodssuchasspeak,walk,
sleep,wake-up,eat,dream,write,read,andsoon.AllthethingsthatIcandocouldbe
seenasmethodsoftheobjectsthatrepresentsme.
So,nowthatyouknowwhatobjectsareandthattheyexposemethodsthatyoucanrun
andpropertiesthatyoucaninspect,you’rereadytostartcoding.Codinginfactissimply
aboutmanagingthoseobjectsthatliveinthesubsetoftheworldthatwe’rereproducingin
oursoftware.Youcancreate,use,reuse,anddeleteobjectsasyouplease.
AccordingtotheDataModelchapterontheofficialPythondocumentation:
“ObjectsarePython’sabstractionfordata.AlldatainaPythonprogramis
representedbyobjectsorbyrelationsbetweenobjects.”
We’lltakeacloserlookatPythonobjectsinChapter6,AdvancedConcepts–OOP,
Decorators,andIterators.Fornow,allweneedtoknowisthateveryobjectinPythonhas
anID(oridentity),atype,andavalue.
WOW! eBook
www.wowebook.org
Oncecreated,theidentityofanobjectisneverchanged.It’sauniqueidentifierforit,and
it’susedbehindthescenesbyPythontoretrievetheobjectwhenwewanttouseit.
Thetypeaswell,neverchanges.Thetypetellswhatoperationsaresupportedbythe
objectandthepossiblevaluesthatcanbeassignedtoit.
We’llseePython’smostimportantdatatypesinChapter2,Built-inDataTypes.
Thevaluecaneitherchangeornot.Ifitcan,theobjectissaidtobemutable,whilewhen
itcannot,theobjectissaidtobeimmutable.
Howdoweuseanobject?Wegiveitanameofcourse!Whenyougiveanobjectaname,
thenyoucanusethenametoretrievetheobjectanduseit.
Inamoregenericsense,objectssuchasnumbers,strings(text),collections,andsoonare
associatedwithaname.Usually,wesaythatthisnameisthenameofavariable.Youcan
seethevariableasbeinglikeabox,whichyoucanusetoholddata.
So,youhavealltheobjectsyouneed:whatnow?Well,weneedtousethem,right?We
maywanttosendthemoveranetworkconnectionorstoretheminadatabase.Maybe
displaythemonawebpageorwritethemintoafile.Inordertodoso,weneedtoreactto
auserfillinginaform,orpressingabutton,oropeningawebpageandperforminga
search.Wereactbyrunningourcode,evaluatingconditionstochoosewhichpartsto
execute,howmanytimes,andunderwhichcircumstances.
Andtodoallthis,basicallyweneedalanguage.That’swhatPythonisfor.Pythonisthe
languagewe’llusetogetherthroughoutthisbooktoinstructthecomputertodosomething
forus.
Now,enoughofthistheoreticalstuff,let’sgetstarted.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
EnterthePython
PythonisthemarvelouscreatureofGuidoVanRossum,aDutchcomputerscientistand
mathematicianwhodecidedtogifttheworldwithaprojecthewasplayingaroundwith
overChristmas1989.Thelanguageappearedtothepublicsomewherearound1991,and
sincethenhasevolvedtobeoneoftheleadingprogramminglanguagesusedworldwide
today.
IstartedprogrammingwhenIwas7yearsold,onaCommodoreVIC20,whichwaslater
replacedbyitsbiggerbrother,theCommodore64.ThelanguagewasBASIC.Lateron,I
landedonPascal,Assembly,C,C++,Java,JavaScript,VisualBasic,PHP,ASP,ASP
.NET,C#,andotherminorlanguagesIcannotevenremember,butonlywhenIlandedon
Python,Ifinallyhadthatfeelingthatyouhavewhenyoufindtherightcouchintheshop.
Whenallofyourbodypartsareyelling,“Buythisone!Thisoneisperfectforus!”
Ittookmeaboutadaytogetusedtoit.ItssyntaxisabitdifferentfromwhatIwasused
to,andingeneral,Iveryrarelyworkedwithalanguagethatdefinesscopingwith
indentation.Butaftergettingpastthatinitialfeelingofdiscomfort(likehavingnew
shoes),Ijustfellinlovewithit.Deeply.Let’sseewhy.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
AboutPython
Beforewegetintothegorydetails,let’sgetasenseofwhysomeonewouldwanttouse
Python(IwouldrecommendyoutoreadthePythonpageonWikipediatogetamore
detailedintroduction).
Tomymind,Pythonexposesthefollowingqualities.
WOW! eBook
www.wowebook.org
Portability
Pythonrunseverywhere,andportingaprogramfromLinuxtoWindowsorMacisusually
justamatteroffixingpathsandsettings.Pythonisdesignedforportabilityandittakes
careofoperatingsystem(OS)specificquirksbehindinterfacesthatshieldyoufromthe
painofhavingtowritecodetailoredtoaspecificplatform.
WOW! eBook
www.wowebook.org
Coherence
Pythonisextremelylogicalandcoherent.Youcanseeitwasdesignedbyabrilliant
computerscientist.Mostofthetimeyoucanjustguesshowamethodiscalled,ifyou
don’tknowit.
Youmaynotrealizehowimportantthisisrightnow,especiallyifyouareatthebeginning,
butthisisamajorfeature.Itmeanslessclutteringinyourhead,lessskimmingthroughthe
documentation,andlessneedformappinginyourbrainwhenyoucode.
WOW! eBook
www.wowebook.org
Developerproductivity
AccordingtoMarkLutz(LearningPython,5thEdition,O’ReillyMedia),aPython
programistypicallyone-fifthtoone-thirdthesizeofequivalentJavaorC++code.This
meansthejobgetsdonefaster.Andfasterisgood.Fastermeansafasterresponseonthe
market.Lesscodenotonlymeanslesscodetowrite,butalsolesscodetoread(and
professionalcodersreadmuchmorethantheywrite),lesscodetomaintain,todebug,and
torefactor.
AnotherimportantaspectisthatPythonrunswithouttheneedoflengthyandtime
consumingcompilationandlinkagesteps,soyoudon’thavetowaittoseetheresultsof
yourwork.
WOW! eBook
www.wowebook.org
Anextensivelibrary
Pythonhasanincrediblywidestandardlibrary(it’ssaidtocomewith“batteries
included”).Ifthatwasn’tenough,thePythoncommunityallovertheworldmaintainsa
bodyofthirdpartylibraries,tailoredtospecificneeds,whichyoucanaccessfreelyatthe
PythonPackageIndex(PyPI).WhenyoucodePythonandyourealizethatyouneeda
certainfeature,inmostcases,thereisatleastonelibrarywherethatfeaturehasalready
beenimplementedforyou.
WOW! eBook
www.wowebook.org
Softwarequality
Pythonisheavilyfocusedonreadability,coherence,andquality.Thelanguageuniformity
allowsforhighreadabilityandthisiscrucialnowadayswherecodeismoreofacollective
effortthanasoloexperience.AnotherimportantaspectofPythonisitsintrinsicmultiparadigmnature.Youcanuseitasscriptinglanguage,butyoualsocanexploitobjectoriented,imperative,andfunctionalprogrammingstyles.Itisversatile.
WOW! eBook
www.wowebook.org
Softwareintegration
AnotherimportantaspectisthatPythoncanbeextendedandintegratedwithmanyother
languages,whichmeansthatevenwhenacompanyisusingadifferentlanguageastheir
mainstreamtool,Pythoncancomeinandactasaglueagentbetweencomplex
applicationsthatneedtotalktoeachotherinsomeway.Thisiskindofanadvancedtopic,
butintherealworld,thisfeatureisveryimportant.
WOW! eBook
www.wowebook.org
Satisfactionandenjoyment
Lastbutnotleast,thefunofit!WorkingwithPythonisfun.Icancodefor8hoursand
leavetheofficehappyandsatisfied,alientothestruggleothercodershavetoendure
becausetheyuselanguagesthatdon’tprovidethemwiththesameamountofwelldesigneddatastructuresandconstructs.Pythonmakescodingfun,nodoubtaboutit.And
funpromotesmotivationandproductivity.
ThesearethemajoraspectswhyIwouldrecommendPythontoeveryonefor.Ofcourse,
therearemanyothertechnicalandadvancedfeaturesthatIcouldhavetalkedabout,but
theydon’treallypertaintoanintroductorysectionlikethisone.Theywillcomeup
naturally,chapterafterchapter,inthisbook.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Whatarethedrawbacks?
Probably,theonlydrawbackthatonecouldfindinPython,whichisnotduetopersonal
preferences,istheexecutionspeed.Typically,Pythonisslowerthanitscompiledbrothers.
ThestandardimplementationofPythonproduces,whenyourunanapplication,a
compiledversionofthesourcecodecalledbytecode(withtheextension.pyc),whichis
thenrunbythePythoninterpreter.Theadvantageofthisapproachisportability,whichwe
payforwithaslowdownduetothefactthatPythonisnotcompileddowntomachine
levelasareotherlanguages.
However,Pythonspeedisrarelyaproblemtoday,henceitswideuseregardlessofthis
suboptimalfeature.Whathappensisthatinreallife,hardwarecostisnolongeraproblem,
andusuallyit’seasyenoughtogainspeedbyparallelizingtasks.Whenitcomesto
numbercrunchingthough,onecanswitchtofasterPythonimplementations,suchasPyPy,
whichprovidesanaverage7-foldspeedupbyimplementingadvancedcompilation
techniques(checkhttp://pypy.org/forreference).
Whendoingdatascience,you’llmostlikelyfindthatthelibrariesthatyouusewith
Python,suchasPandasandNumpy,achievenativespeedduetothewaytheyare
implemented.
Ifthatwasn’tagoodenoughargument,youcanalwaysconsiderthatPythonisdrivingthe
backendofservicessuchasSpotifyandInstagram,whereperformanceisaconcern.
Nonetheless,Pythondoesitsjobperfectlyadequately.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
WhoisusingPythontoday?
Notyetconvinced?Let’stakeaverybrieflookatthecompaniesthatareusingPython
today:Google,YouTube,Dropbox,Yahoo,ZopeCorporation,IndustrialLight&Magic,
WaltDisneyFeatureAnimation,Pixar,NASA,NSA,RedHat,Nokia,IBM,Netflix,Yelp,
Intel,Cisco,HP,Qualcomm,andJPMorganChase,justtonameafew.
EvengamessuchasBattlefield2,Civilization4,andQuArKareimplementedusing
Python.
Pythonisusedinmanydifferentcontexts,suchassystemprogramming,web
programming,GUIapplications,gamingandrobotics,rapidprototyping,system
integration,datascience,databaseapplications,andmuchmore.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Settinguptheenvironment
BeforewetalkaboutinstallingPythononyoursystem,letmetellyouaboutwhichPython
versionI’llbeusinginthisbook.
WOW! eBook
www.wowebook.org
Python2versusPython3–thegreatdebate
Pythoncomesintwomainversions—Python2,whichisthepast—andPython3,whichis
thepresent.Thetwoversions,thoughverysimilar,areincompatibleonsomeaspects.
Intherealworld,Python2isactuallyquitefarfrombeingthepast.Inshort,eventhough
Python3hasbeenoutsince2008,thetransitionphaseisstillfarfrombeingover.Thisis
mostlyduetothefactthatPython2iswidelyusedintheindustry,andofcourse,
companiesaren’tsokeenonupdatingtheirsystemsjustforthesakeofupdating,
followingtheifitain’tbroke,don’tfixitphilosophy.Youcanreadallaboutthetransition
betweenthetwoversionsontheWeb.
Anotherissuethatwashinderingthetransitionistheavailabilityofthird-partylibraries.
Usually,aPythonprojectreliesontensofexternallibraries,andofcourse,whenyoustart
anewproject,youneedtobesurethatthereisalreadyaversion3compatiblelibraryfor
anybusinessrequirementthatmaycomeup.Ifthat’snotthecase,startingabrandnew
projectinPython3meansintroducingapotentialrisk,whichmanycompaniesarenot
happytotake.
Atthetimeofwriting,themajorityofthemostwidelyusedlibrarieshavebeenportedto
Python3,andit’squitesafetostartaprojectinPython3formostcases.Manyofthe
librarieshavebeenrewrittensothattheyarecompatiblewithbothversions,mostly
harnessingthepowerofthesix(2x3)library,whichhelpsintrospectingandadaptingthe
behavioraccordingtotheversionused.
OnmyLinuxbox(Ubuntu14.04),IhavethefollowingPythonversion:
>>>importsys
>>>print(sys.version)
3.4.0(default,Apr112014,13:05:11)
[GCC4.8.2]
SoyoucanseethatmyPythonversionis3.4.0.TheprecedingtextisalittlebitofPython
codethatItypedintomyconsole.We’lltalkaboutitinamoment.
AlltheexamplesinthisbookwillberunusingthisPythonversion.Mostofthemwillrun
alsoinPython2(Ihaveversion2.7.6installedaswell),andthosethatwon’twilljust
requiresomeminoradjustmentstocaterforthesmallincompatibilitiesbetweenthetwo
versions.AnotherreasonbehindthischoiceisthatIthinkit’sbettertolearnPython3,and
then,ifyouneedto,learnthedifferencesithaswithPython2,ratherthangoingtheother
wayaround.
Don’tworryaboutthisversionthingthough:it’snotthatbiganissueinpractice.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
InstallingPython
Ineverreallygotthepointofhavingasetupsectioninabook,regardlessofwhatitisthat
youhavetosetup.Mostofthetime,betweenthetimetheauthorwritestheinstruction
andthetimeyouactuallytrythemout,monthshavepassed.Thatis,ifyou’relucky.One
versionchangeandthingsmaynotworkthewayitisdescribedinthebook.Luckily,we
havetheWebnow,soinordertohelpyougetupandrunning,I’lljustgiveyoupointers
andobjectives.
Tip
IfanyoftheURLsorresourcesI’llpointyoutoarenolongertherebythetimeyouread
thisbook,justremember:Googleisyourfriend.
WOW! eBook
www.wowebook.org
SettingupthePythoninterpreter
Firstofall,let’stalkaboutyourOS.Pythonisfullyintegratedandmostlikelyalready
installedinbasicallyalmosteveryLinuxdistribution.IfyouhaveaMac,it’slikelythat
Pythonisalreadythereaswell(however,possiblyonlyPython2.7),whereasifyou’re
usingWindows,youprobablyneedtoinstallit.
GettingPythonandthelibrariesyouneedupandrunningrequiresabitofhandiwork.
LinuxhappenstobethemostuserfriendlyOSforPythonprogrammers,Windowsonthe
otherhandistheonethatrequiresthebiggesteffort,Macbeingsomewhereinbetween.
Forthisreason,ifyoucanchoose,IsuggestyoutouseLinux.Ifyoucan’t,andyouhavea
Mac,thengoforitanyway.IfyouuseWindows,you’llbefinefortheexamplesinthis
book,butingeneralworkingwithPythonwillrequireyouabitmoretweaking.
MyOSisUbuntu14.04,andthisiswhatIwillusethroughoutthebook,alongwith
Python3.4.0.
TheplaceyouwanttostartistheofficialPythonwebsite:https://www.python.org.This
websitehoststheofficialPythondocumentationandmanyotherresourcesthatyouwill
findveryuseful.Takethetimetoexploreit.
Tip
Anotherexcellent,resourcefulwebsiteonPythonanditsecosystemishttp://docs.pythonguide.org.
FindthedownloadsectionandchoosetheinstallerforyourOS.IfyouareonWindows,
makesurethatwhenyouruntheinstaller,youchecktheoptioninstallpip(actually,I
wouldsuggesttomakeacompleteinstallation,justtobesafe,ofallthecomponentsthe
installerholds).We’lltalkaboutpiplater.
NowthatPythonisinstalledinyoursystem,theobjectiveistobeabletoopenaconsole
andrunthePythoninteractiveshellbytypingpython.
Note
PleasenotethatIusuallyrefertothePythoninteractiveshellsimplyasPythonconsole.
ToopentheconsoleinWindows,gototheStartmenu,chooseRun,andtypecmd.Ifyou
encounteranythingthatlookslikeapermissionproblemwhileworkingontheexamples
ofthisbook,pleasemakesureyouarerunningtheconsolewithadministratorrights.
OntheMacOSX,youcanstartaterminalbygoingtoApplications|Utilities|
Terminal.
IfyouareonLinux,youknowallthatthereistoknowabouttheconsole.
Note
IwillusethetermconsoleinterchangeablytoindicatetheLinuxconsole,theWindows
commandprompt,andtheMacterminal.Iwillalsoindicatethecommand-lineprompt
withtheLinuxdefaultformat,likethis:
WOW! eBook
www.wowebook.org
$sudoapt-getupdate
Whateverconsoleyouopen,typepythonattheprompt,andmakesurethePython
interactiveshellshowsup.Typeexit()toquit.Keepinmindthatyoumayhaveto
specifypython3ifyourOScomeswithPython2.*preinstalled.
ThisishowitshouldlookonWindows7:
AndthisishowitshouldlookonLinux:
NowthatPythonissetupandyoucanrunit,it’stimetomakesureyouhavetheothertool
thatwillbeindispensabletofollowtheexamplesinthebook:virtualenv.
WOW! eBook
www.wowebook.org
Aboutvirtualenv
Asyouprobablyhaveguessedbyitsname,virtualenvisallaboutvirtualenvironments.
Letmeexplainwhattheyareandwhyweneedthemandletmedoitbymeansofasimple
example.
YouinstallPythononyoursystemandyoustartworkingonawebsiteforclientX.You
createaprojectfolderandstartcoding.Alongthewayyoualsoinstallsomelibraries,for
exampletheDjangoframework,whichwe’llseeindepthinChapter10,Web
DevelopmentDoneRight.Let’ssaytheDjangoversionyouinstallforprojectXis1.7.1.
Now,yourwebsiteissogoodthatyougetanotherclient,Y.Hewantsyoutobuildanother
website,soyoustartprojectYand,alongtheway,youneedtoinstallDjangoagain.The
onlyissueisthatnowtheDjangoversionis1.8andyoucannotinstallitonyoursystem
becausethiswouldreplacetheversionyouinstalledforprojectX.Youdon’twanttorisk
introducingincompatibilityissues,soyouhavetwochoices:eitheryoustickwiththe
versionyouhavecurrentlyonyourmachine,oryouupgradeitandmakesurethefirst
projectisstillfullyworkingcorrectlywiththenewversion.
Let’sbehonest,neitheroftheseoptionsisveryappealing,right?Definitelynot.So,here’s
thesolution:virtualenv!
virtualenvisatoolthatallowsyoutocreateavirtualenvironment.Inotherwords,itisa
tooltocreateisolatedPythonenvironments,eachofwhichisafolderthatcontainsallthe
necessaryexecutablestousethepackagesthataPythonprojectwouldneed(thinkof
packagesaslibrariesforthetimebeing).
SoyoucreateavirtualenvironmentforprojectX,installallthedependencies,andthen
youcreateavirtualenvironmentforprojectY,installingallitsdependencieswithoutthe
slightestworrybecauseeverylibraryyouinstallendsupwithintheboundariesofthe
appropriatevirtualenvironment.Inourexample,projectXwillholdDjango1.7.1,while
projectYwillholdDjango1.8.
Note
Itisofvitalimportancethatyouneverinstalllibrariesdirectlyatthesystemlevel.Linux
forexamplereliesonPythonformanydifferenttasksandoperations,andifyoufiddle
withthesysteminstallationofPython,youriskcompromisingtheintegrityofthewhole
system(guesstowhomthishappened…).Sotakethisasarule,suchasbrushingyour
teethbeforegoingtobed:always,alwayscreateavirtualenvironmentwhenyoustarta
newproject.
Toinstallvirtualenvonyoursystem,thereareafewdifferentways.OnaDebian-based
distributionofLinuxforexample,youcaninstallitwiththefollowingcommand:
$sudoapt-getinstallpython-virtualenv
Probably,theeasiestwayistousepipthough,withthefollowingcommand:
$sudopipinstallvirtualenv#sudomaybyoptional
WOW! eBook
www.wowebook.org
pipisapackagemanagementsystemusedtoinstallandmanagesoftwarepackages
writteninPython.
Python3hasbuilt-insupportforvirtualenvironments,butinpractice,theexternal
librariesarestillthedefaultonproductionsystems.Ifyouhavetroublegettingvirtualenv
upandrunning,pleaserefertothevirtualenvofficialwebsite:https://virtualenv.pypa.io.
WOW! eBook
www.wowebook.org
Yourfirstvirtualenvironment
Itisveryeasytocreateavirtualenvironment,butaccordingtohowyoursystemis
configuredandwhichPythonversionyouwantthevirtualenvironmenttorun,youneedto
runthecommandproperly.Anotherthingyouwillneedtodowithavirtualenv,whenyou
wanttoworkwithit,istoactivateit.Activatingavirtualenvbasicallyproducessomepath
jugglingbehindthescenessothatwhenyoucallthePythoninterpreter,you’reactually
callingtheactivevirtualenvironmentone,insteadofthemeresystemone.
I’llshowyouafullexampleonbothLinuxandWindows.Wewill:
1. Createafoldernamedlearning.pythonunderyourprojectroot(whichinmycaseis
afoldercalledsrv,inmyhomefolder).Pleaseadaptthepathsaccordingtothesetup
youfancyonyourbox.
2. Withinthelearning.pythonfolder,wewillcreateavirtualenvironmentcalled
.lpvenv.
Note
Somedevelopersprefertocallallvirtualenvironmentsusingthesamename(for
example,.venv).Thiswaytheycanrunscriptsagainstanyvirtualenvbyjust
knowingthenameoftheprojecttheydwellin.Thisisaverycommontechniquethat
Iuseaswell.Thedotin.venvisbecauseinLinux/Macprependinganamewitha
dotmakesthatfileorfolderinvisible.
3. Aftercreatingthevirtualenvironment,wewillactivateit(thisisslightlydifferent
betweenLinux,Mac,andWindows).
4. Then,we’llmakesurethatwearerunningthedesiredPythonversion(3.4.*)by
runningthePythoninteractiveshell.
5. Finally,wewilldeactivatethevirtualenvironmentusingthedeactivatecommand.
Thesefivesimplestepswillshowyouallyouhavetodotostartanduseaproject.
Here’sanexampleofhowthosestepsmightlooklikeonLinux(commandsthatstartwith
a#arecomments):
WOW! eBook
www.wowebook.org
NoticethatIhadtoexplicitlytellvirtualenvtousethePython3.4interpreterbecauseon
myboxPython2.7isthedefaultone.HadInotdonethat,Iwouldhavehadavirtual
environmentwithPython2.7insteadofPython3.4.
Youcancombinethetwoinstructionsforstep2inonesinglecommandlikethis:
$virtualenv-p$(whichpython3.4).lpvenv
Ipreferredtobeexplicitlyverboseinthisinstance,tohelpyouunderstandeachbitofthe
procedure.
Anotherthingtonoticeisthatinordertoactivateavirtualenvironment,weneedtorun
the/bin/activatescript,whichneedstobesourced(whenascriptis“sourced”,itmeans
thatitseffectsstickaroundwhenit’sdonerunning).Thisisveryimportant.Alsonotice
howthepromptchangesafterweactivatethevirtualenvironment,showingitsnameon
theleft(andhowitdisappearswhenwedeactivate).InMacOS,thestepsarethesameso
Iwon’trepeatthemhere.
Nowlet’shavealookathowwecanachievethesameresultinWindows.Youwill
probablyhavetoplayaroundabit,especiallyifyouhaveadifferentWindowsorPython
versionthanI’musinghere.Thisisallgoodexperiencethough,sotryandthinkpositively
attheinitialstrugglethateverycoderhastogothroughinordertogetthingsgoing.
WOW! eBook
www.wowebook.org
Here’showitshouldlookonWindows(commandsthatstartwith::arecomments):
NoticethereareafewsmalldifferencesfromtheLinuxversion.Apartfromthe
commandstocreateandnavigatethefolders,oneimportantdifferenceishowyouactivate
yourvirtualenv.Also,inWindowsthereisnowhichcommand,soweusedthewhere
command.
Atthispoint,youshouldbeabletocreateandactivateavirtualenvironment.Pleasetry
andcreateanotheronewithoutmeguidingyou,getacquaintedtothisprocedurebecause
it’ssomethingthatyouwillalwaysbedoing:weneverworksystem-widewithPython,
remember?It’sextremelyimportant.
So,withthescaffoldingoutoftheway,we’rereadytotalkabitmoreaboutPythonand
howyoucanuseit.Beforewedoitthough,allowmetospendafewwordsaboutthe
console.
WOW! eBook
www.wowebook.org
Yourfriend,theconsole
InthiseraofGUIsandtouchscreendevices,itseemsalittleridiculoustohavetoresortto
atoolsuchastheconsole,wheneverythingisjustaboutoneclickaway.
Butthetruthiseverytimeyouremoveyourrighthandfromthekeyboard(ortheleftone,
ifyou’realefty)tograbyourmouseandmovethecursorovertothespotyouwantto
click,you’relosingtime.Gettingthingsdonewiththeconsole,counter-intuitivelyasit
maybe,resultsinhigherproductivityandspeed.Iknow,youhavetotrustmeonthis.
Speedandproductivityareimportantandpersonally,Ihavenothingagainstthemouse,
butthereisanotherverygoodreasonforwhichyoumaywanttogetwellacquaintedwith
theconsole:whenyoudevelopcodethatendsuponsomeserver,theconsolemightbethe
onlyavailabletool.Ifyoumakefriendswithit,Ipromiseyou,youwillnevergetlost
whenit’sofutmostimportancethatyoudon’t(typically,whenthewebsiteisdownand
youhavetoinvestigateveryquicklywhat’sgoingon).
Soit’sreallyuptoyou.Ifyou’reindoubt,pleasegrantmethebenefitofthedoubtand
giveitatry.It’seasierthanyouthink,andyou’llneverregretit.Thereisnothingmore
pitifulthanagooddeveloperwhogetslostwithinanSSHconnectiontoaserverbecause
theyareusedtotheirowncustomsetoftools,andonlytothat.
Now,let’sgetbacktoPython.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
HowyoucanrunaPythonprogram
ThereareafewdifferentwaysinwhichyoucanrunaPythonprogram.
WOW! eBook
www.wowebook.org
RunningPythonscripts
Pythoncanbeusedasascriptinglanguage.Infact,italwaysprovesitselfveryuseful.
Scriptsarefiles(usuallyofsmalldimensions)thatyounormallyexecutetodosomething
likeatask.Manydevelopersenduphavingtheirownarsenaloftoolsthattheyfirewhen
theyneedtoperformatask.Forexample,youcanhavescriptstoparsedatainaformat
andrenderitintoanotherdifferentformat.Oryoucanuseascripttoworkwithfilesand
folders.Youcancreateormodifyconfigurationfiles,andmuchmore.Technically,thereis
notmuchthatcannotbedoneinascript.
It’squitecommontohavescriptsrunningataprecisetimeonaserver.Forexample,if
yourwebsitedatabaseneedscleaningevery24hours(forexample,thetablethatstoresthe
usersessions,whichexpireprettyquicklybutaren’tcleanedautomatically),youcouldset
upacronjobthatfiresyourscriptat3:00A.M.everyday.
Note
AccordingtoWikipedia,thesoftwareutilityCronisatime-basedjobschedulerinUnixlikecomputeroperatingsystems.Peoplewhosetupandmaintainsoftwareenvironments
usecrontoschedulejobs(commandsorshellscripts)torunperiodicallyatfixedtimes,
dates,orintervals.
IhavePythonscriptstodoallthemenialtasksthatwouldtakememinutesormoretodo
manually,andatsomepoint,Idecidedtoautomate.Forexample,Ihavealaptopthat
doesn’thaveaFnkeytotogglethetouchpadonandoff.Ifindthisveryannoying,andI
don’twanttogoclickingaboutthroughseveralmenuswhenIneedtodoit,soIwrotea
smallscriptthatissmartenoughtotellmysystemtotogglethetouchpadactivestate,and
nowIcandoitwithonesimpleclickfrommylauncher.Priceless.
We’lldevotehalfofChapter8,TheEdges–GUIsandScriptsonscriptingwithPython.
WOW! eBook
www.wowebook.org
RunningthePythoninteractiveshell
AnotherwayofrunningPythonisbycallingtheinteractiveshell.Thisissomethingwe
alreadysawwhenwetypedpythononthecommandlineofourconsole.
Soopenaconsole,activateyourvirtualenvironment(whichbynowshouldbesecond
naturetoyou,right?),andtypepython.Youwillbepresentedwithacoupleoflinesthat
shouldlooklikethis(ifyouareonLinux):
Python3.4.0(default,Apr112014,13:05:11)
[GCC4.8.2]onlinux
Type"help","copyright","credits"or"license"formoreinformation.
Those>>>arethepromptoftheshell.TheytellyouthatPythoniswaitingforyoutotype
something.Ifyoutypeasimpleinstruction,somethingthatfitsinoneline,that’sallyou’ll
see.However,ifyoutypesomethingthatrequiresmorethanonelineofcode,theshell
willchangethepromptto...,givingyouavisualcluethatyou’retypingamultiline
statement(oranythingthatwouldrequiremorethanonelineofcode).
Goon,tryitout,let’sdosomebasicmaths:
>>>2+4
6
>>>10/4
2.5
>>>2**1024
179769313486231590772930519078902473361797697894230657273430081157732675805
500963132708477322407536021120113879871393357658789768814416622492847430639
474124377767893424865485276302219601246094119453082952085005768838150682342
462881473913110540827237163350510684586298239947245938479716304835356329624
224137216
Thelastoperationisshowingyousomethingincredible.Weraise2tothepowerof1024,
andPythonishandlingthistaskwithnotroubleatall.TrytodoitinJava,C++,orC#.It
won’twork,unlessyouusespeciallibrariestohandlesuchbignumbers.
Iusetheinteractiveshelleveryday.It’sextremelyusefultodebugveryquickly,for
example,tocheckifadatastructuresupportsanoperation.Ormaybetoinspectorruna
pieceofcode.
WhenyouuseDjango(awebframework),theinteractiveshelliscoupledwithitand
allowsyoutoworkyourwaythroughtheframeworktools,toinspectthedatainthe
database,andmanymorethings.Youwillfindthattheinteractiveshellwillsoonbecome
oneofyourdearestfriendsonthejourneyyouareembarkingon.
Anothersolution,whichcomesinamuchnicergraphiclayout,istouseIDLE
(IntegratedDeveLopmentEnvironment).It’squiteasimpleIDE,whichisintended
mostlyforbeginners.Ithasaslightlylargersetofcapabilitiesthanthenakedinteractive
shellyougetintheconsole,soyoumaywanttoexploreit.Itcomesforfreeinthe
WindowsPythoninstallerandyoucaneasilyinstallitinanyothersystem.Youcanfind
informationaboutitonthePythonwebsite.
WOW! eBook
www.wowebook.org
GuidoVanRossumnamedPythonaftertheBritishcomedygroupMontyPython,soit’s
rumoredthatthenameIDLEhasbeenchoseninhonorofErikIdle,oneofMonty
Python’sfoundingmembers.
WOW! eBook
www.wowebook.org
RunningPythonasaservice
Apartfrombeingrunasascript,andwithintheboundariesofashell,Pythoncanbecoded
andrunaspropersoftware.We’llseemanyexamplesthroughoutthebookaboutthis
mode.Andwe’llunderstandmoreaboutitinamoment,whenwe’lltalkabouthow
Pythoncodeisorganizedandrun.
WOW! eBook
www.wowebook.org
RunningPythonasaGUIapplication
PythoncanalsoberunasaGUI(GraphicalUserInterface).Thereareseveral
frameworksavailable,someofwhicharecross-platformandsomeothersareplatformspecific.InChapter8,TheEdges–GUIsandScripts,we’llseeanexampleofaGUI
applicationcreatedusingTkinter,whichisanobject-orientedlayerthatlivesontopofTk
(TkintermeansTkInterface).
Note
Tkisagraphicaluserinterfacetoolkitthattakesdesktopapplicationdevelopmenttoa
higherlevelthantheconventionalapproach.ItisthestandardGUIforTcl(Tool
CommandLanguage),butalsoformanyotherdynamiclanguagesandcanproducerich
nativeapplicationsthatrunseamlesslyunderWindows,Linux,MacOSX,andmore.
TkintercomesbundledwithPython,thereforeitgivestheprogrammereasyaccesstothe
GUIworld,andforthesereasons,IhavechosenittobetheframeworkfortheGUI
examplesthatI’llpresentinthisbook.
AmongtheotherGUIframeworks,wefindthatthefollowingarethemostwidelyused:
PyQt
wxPython
PyGtk
Describingthemindetailisoutsidethescopeofthisbook,butyoucanfindallthe
informationyouneedonthePythonwebsiteintheGUIProgrammingsection.IfGUIsare
whatyou’relookingfor,remembertochoosetheoneyouwantaccordingtosome
principles.Makesurethey:
Offerallthefeaturesyoumayneedtodevelopyourproject
Runonalltheplatformsyoumayneedtosupport
Relyonacommunitythatisaswideandactiveaspossible
Wrapgraphicdrivers/toolsthatyoucaneasilyinstall/access
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
HowisPythoncodeorganized
Let’stalkalittlebitabouthowPythoncodeisorganized.Inthisparagraph,we’llstart
goingdowntherabbitholealittlebitmoreandintroduceabitmoretechnicalnamesand
concepts.
Startingwiththebasics,howisPythoncodeorganized?Ofcourse,youwriteyourcode
intofiles.Whenyousaveafilewiththeextension.py,thatfileissaidtobeaPython
module.
Tip
Ifyou’reonWindowsorMac,whichtypicallyhidefileextensionstotheuser,please
makesureyouchangetheconfigurationsothatyoucanseethecompletenameofthe
files.Thisisnotstrictlyarequirement,butaheartysuggestion.
Itwouldbeimpracticaltosaveallthecodethatitisrequiredforsoftwaretoworkwithin
onesinglefile.Thatsolutionworksforscripts,whichareusuallynotlongerthanafew
hundredlines(andoftentheyarequiteshorterthanthat).
AcompletePythonapplicationcanbemadeofhundredsofthousandsoflinesofcode,so
youwillhavetoscatteritthroughdifferentmodules.Better,butnotnearlygoodenough.It
turnsoutthatevenlikethisitwouldstillbeimpracticaltoworkwiththecode.SoPython
givesyouanotherstructure,calledpackage,whichallowsyoutogroupmodulestogether.
Apackageisnothingmorethanafolder,whichmustcontainaspecialfile,__init__.py
thatdoesn’tneedtoholdanycodebutwhosepresenceisrequiredtotellPythonthatthe
folderisnotjustsomefolder,butit’sactuallyapackage(notethatasofPython3.3
__init__.pyisnotstrictlyrequiredanymore).
Asalways,anexamplewillmakeallofthismuchclearer.Ihavecreatedanexample
structureinmybookproject,andwhenItypeinmyLinuxconsole:
$tree-vexample
Igetatreerepresentationofthecontentsofthech1/examplefolder,whichholdsthecode
fortheexamplesofthischapter.Here’showastructureofarealsimpleapplicationcould
looklike:
example/
├──core.py
├──run.py
└──util
├──__init__.py
├──db.py
├──math.py
└──network.py
Youcanseethatwithintherootofthisexample,wehavetwomodules,core.pyand
run.py,andonepackage:util.Withincore.py,theremaybethecorelogicofour
application.Ontheotherhand,withintherun.pymodule,wecanprobablyfindthelogic
tostarttheapplication.Withintheutilpackage,Iexpecttofindvariousutilitytools,and
WOW! eBook
www.wowebook.org
infact,wecanguessthatthemodulestherearecalledbythetypeoftoolstheyhold:
db.pywouldholdtoolstoworkwithdatabases,math.pywouldofcoursehold
mathematicaltools(maybeourapplicationdealswithfinancialdata),andnetwork.py
wouldprobablyholdtoolstosend/receivedataonnetworks.
Asexplainedbefore,the__init__.pyfileistherejusttotellPythonthatutilisa
packageandnotjustamerefolder.
Hadthissoftwarebeenorganizedwithinmodulesonly,itwouldhavebeenmuchharderto
inferitsstructure.Iputamoduleonlyexampleunderthech1/files_onlyfolder,seeitfor
yourself:
$tree-vfiles_only
Thisshowsusacompletelydifferentpicture:
files_only/
├──core.py
├──db.py
├──math.py
├──network.py
└──run.py
Itisalittlehardertoguesswhateachmoduledoes,right?Now,considerthatthisisjusta
simpleexample,soyoucanguesshowmuchharderitwouldbetounderstandareal
applicationifwecouldn’torganizethecodeinpackagesandmodules.
WOW! eBook
www.wowebook.org
Howdoweusemodulesandpackages
Whenadeveloperiswritinganapplication,itisverylikelythattheywillneedtoapply
thesamepieceoflogicindifferentpartsofit.Forexample,whenwritingaparserforthe
datathatcomesfromaformthatausercanfillinawebpage,theapplicationwillhaveto
validatewhetheracertainfieldisholdinganumberornot.Regardlessofhowthelogicfor
thiskindofvalidationiswritten,it’sverylikelythatitwillbeneededinmorethanone
place.Forexampleinapollapplication,wheretheuserisaskedmanyquestion,it’slikely
thatseveralofthemwillrequireanumericanswer.Forexample:
Whatisyourage
Howmanypetsdoyouown
Howmanychildrendoyouhave
Howmanytimeshaveyoubeenmarried
Itwouldbeverybadpracticetocopypaste(or,moreproperlysaid:duplicate)the
validationlogicineveryplacewhereweexpectanumericanswer.Thiswouldviolatethe
DRY(Don’tRepeatYourself)principle,whichstatesthatyoushouldneverrepeatthe
samepieceofcodemorethanonceinyourapplication.Ifeeltheneedtostressthe
importanceofthisprinciple:youshouldneverrepeatthesamepieceofcodemorethan
onceinyourapplication(gottheirony?).
Thereareseveralreasonswhyrepeatingthesamepieceoflogiccanbeverybad,themost
importantonesbeing:
Therecouldbeabuginthelogic,andtherefore,youwouldhavetocorrectitinevery
placethatlogicisapplied.
Youmaywanttoamendthewayyoucarryoutthevalidation,andagainyouwould
havetochangeitineveryplaceitisapplied.
Youmayforgettofix/amendapieceoflogicbecauseyoumisseditwhensearching
forallitsoccurrences.Thiswouldleavewrong/inconsistentbehaviorinyour
application.
Yourcodewouldbelongerthanneeded,fornogoodreason.
Pythonisawonderfullanguageandprovidesyouwithallthetoolsyouneedtoapplyall
thecodingbestpractices.Forthisparticularexample,weneedtobeabletoreuseapiece
ofcode.Tobeabletoreuseapieceofcode,weneedtohaveaconstructthatwillholdthe
codeforussothatwecancallthatconstructeverytimeweneedtorepeatthelogicinside
it.Thatconstructexists,andit’scalledfunction.
I’mnotgoingtoodeepintothespecificshere,sopleasejustrememberthatafunctionisa
blockoforganized,reusablecodewhichisusedtoperformatask.Functionscanassume
manyformsandnames,accordingtowhatkindofenvironmenttheybelongto,butfor
nowthisisnotimportant.We’llseethedetailswhenweareabletoappreciatethem,later
on,inthebook.Functionsarethebuildingblocksofmodularityinyourapplication,and
theyarealmostindispensable(unlessyou’rewritingasupersimplescript,you’lluse
functionsallthetime).We’llexplorefunctionsinChapter4,Functions,theBuilding
BlocksofCode.
WOW! eBook
www.wowebook.org
Pythoncomeswithaveryextensivelibrary,asIalreadysaidafewpagesago.Now,
maybeit’sagoodtimetodefinewhatalibraryis:alibraryisacollectionoffunctions
andobjectsthatprovidefunctionalitiesthatenrichtheabilitiesofalanguage.
Forexample,withinPython’smathlibrarywecanfindaplethoraoffunctions,oneof
whichisthefactorialfunction,whichofcoursecalculatesthefactorialofanumber.
Note
Inmathematics,thefactorialofanon-negativeintegernumberN,denotedasN!,is
definedastheproductofallpositiveintegerslessthanorequaltoN.Forexample,the
factorialof5iscalculatedas:
5!=5*4*3*2*1=120
Thefactorialof0is0!=1,torespecttheconventionforanemptyproduct.
So,ifyouwantedtousethisfunctioninyourcode,allyouwouldhavetodoistoimportit
andcallitwiththerightinputvalues.Don’tworrytoomuchifinputvaluesandthe
conceptofcallingisnotveryclearfornow,pleasejustconcentrateontheimportpart.
Note
Weusealibrarybyimportingwhatweneedfromit,andthenweuseit.
InPython,tocalculatethefactorialofnumber5,wejustneedthefollowingcode:
>>>frommathimportfactorial
>>>factorial(5)
120
Note
Whateverwetypeintheshell,ifithasaprintablerepresentation,willbeprintedonthe
consoleforus(inthiscase,theresultofthefunctioncall:120).
So,let’sgobacktoourexample,theonewithcore.py,run.py,util,andsoon.
Inourexample,thepackageutilisourutilitylibrary.Ourcustomutilitybeltthatholds
allthosereusabletools(thatis,functions),whichweneedinourapplication.Someof
themwilldealwithdatabases(db.py),somewiththenetwork(network.py),andsome
willperformmathematicalcalculations(math.py)thatareoutsidethescopeofPython’s
standardmathlibraryandtherefore,wehadtocodethemforourselves.
Wewillseeindetailhowtoimportfunctionsandusethemintheirdedicatedchapter.
Let’snowtalkaboutanotherveryimportantconcept:Python’sexecutionmodel.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Python’sexecutionmodel
Inthisparagraph,Iwouldliketointroduceyoutoafewveryimportantconcepts,suchas
scope,names,andnamespaces.YoucanreadallaboutPython’sexecutionmodelinthe
officialLanguagereference,ofcourse,butIwouldarguethatitisquitetechnicaland
abstract,soletmegiveyoualessformalexplanationfirst.
WOW! eBook
www.wowebook.org
Namesandnamespaces
Sayyouarelookingforabook,soyougotothelibraryandasksomeoneforthebookyou
wanttofetch.Theytellyousomethinglike“secondfloor,sectionX,rowthree”.Soyou
goupthestairs,lookforsectionX,andsoon.
Itwouldbeverydifferenttoenteralibrarywhereallthebooksarepiledtogetherin
randomorderinonebigroom.Nofloors,nosections,norows,noorder.Fetchingabook
wouldbeextremelyhard.
Whenwewritecodewehavethesameissue:wehavetotryandorganizeitsothatitwill
beeasyforsomeonewhohasnopriorknowledgeaboutittofindwhatthey’relookingfor.
Whensoftwareisstructuredcorrectly,italsopromotescodereuse.Ontheotherhand,
disorganizedsoftwareismorelikelytoexposescatteredpiecesofduplicatedlogic.
Firstofall,let’sstartwiththebook.WerefertoabookbyitstitleandinPythonlingo,that
wouldbeaname.Pythonnamesaretheclosestabstractiontowhatotherlanguagescall
variables.Namesbasicallyrefertoobjectsandareintroducedbynamebindingoperations.
Let’smakeaquickexample(noticethatanythingthatfollowsa#isacomment):
>>>n=3#integernumber
>>>address="221bBakerStreet,NW16XE,London"#S.Holmes
>>>employee={
...'age':45,
...'role':'CTO',
...'SSN':'AB1234567',
...}
>>>#let'sprintthem
>>>n
3
>>>address
'221bBakerStreet,NW16XE,London'
>>>employee
{'role':'CTO','SSN':'AB1234567','age':45}
>>>#whatifItrytoprintanameIdidn'tdefine?
>>>other_name
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
NameError:name'other_name'isnotdefined
Wedefinedthreeobjectsintheprecedingcode(doyourememberwhatarethethree
featureseveryPythonobjecthas?):
Anintegernumbern(type:int,value:3)
Astringaddress(type:str,value:SherlockHolmes’address)
Adictionaryemployee(type:dict,value:adictionarywhichholdsthreekey/value
pairs)
Don’tworry,Iknowyou’renotsupposedtoknowwhatadictionaryis.We’llseeinthe
nextchapterthatit’sthekingofPythondatastructures.
Tip
WOW! eBook
www.wowebook.org
Haveyounoticedthatthepromptchangedfrom>>>to...whenItypedinthedefinition
ofemployee?That’sbecausethedefinitionspansovermultiplelines.
So,whataren,addressandemployee?Theyarenames.Namesthatwecanuseto
retrievedatawithinourcode.Theyneedtobekeptsomewheresothatwheneverweneed
toretrievethoseobjects,wecanusetheirnamestofetchthem.Weneedsomespaceto
holdthem,hence:namespaces!
Anamespaceisthereforeamappingfromnamestoobjects.Examplesarethesetofbuiltinnames(containingfunctionsthatarealwaysaccessibleforfreeinanyPythonprogram),
theglobalnamesinamodule,andthelocalnamesinafunction.Eventhesetofattributes
ofanobjectcanbeconsideredanamespace.
Thebeautyofnamespacesisthattheyallowyoutodefineandorganizeyournameswith
clarity,withoutoverlappingorinterference.Forexample,thenamespaceassociatedwith
thatbookwewerelookingforinthelibrarycanbeusedtoimportthebookitself,likethis:
fromlibrary.second_floor.section_x.row_threeimportbook
Westartfromthelibrarynamespace,andbymeansofthedot(.)operator,wewalkinto
thatnamespace.Withinthisnamespace,welookforsecond_floor,andagainwewalk
intoitwiththe.operator.Wethenwalkintosection_x,andfinallywithinthelast
namespace,row_tree,wefindthenamewewerelookingfor:book.
Walkingthroughanamespacewillbeclearerwhenwe’llbedealingwithrealcode
examples.Fornow,justkeepinmindthatnamespacesareplaceswherenamesare
associatedtoobjects.
Thereisanotherconcept,whichiscloselyrelatedtothatofanamespace,whichI’dliketo
brieflytalkabout:thescope.
WOW! eBook
www.wowebook.org
Scopes
AccordingtoPython’sdocumentation,ascopeisatextualregionofaPythonprogram,
whereanamespaceisdirectlyaccessible.Directlyaccessiblemeansthatwhenyou’re
lookingforanunqualifiedreferencetoaname,Pythontriestofinditinthenamespace.
Scopesaredeterminedstatically,butactuallyduringruntimetheyareuseddynamically.
Thismeansthatbyinspectingthesourcecodeyoucantellwhatthescopeofanobjectis,
butthisdoesn’tpreventthesoftwaretoalterthatduringruntime.Therearefourdifferent
scopesthatPythonmakesaccessible(notnecessarilyallofthempresentatthesametime,
ofcourse):
Thelocalscope,whichistheinnermostoneandcontainsthelocalnames.
Theenclosingscope,thatis,thescopeofanyenclosingfunction.Itcontainsnonlocalnamesandalsonon-globalnames.
Theglobalscopecontainstheglobalnames.
Thebuilt-inscopecontainsthebuilt-innames.Pythoncomeswithasetoffunctions
thatyoucanuseinaoff-the-shelffashion,suchasprint,all,abs,andsoon.They
liveinthebuilt-inscope.
Theruleisthefollowing:whenwerefertoaname,Pythonstartslookingforitinthe
currentnamespace.Ifthenameisnotfound,Pythoncontinuesthesearchtotheenclosing
scopeandthiscontinueuntilthebuilt-inscopeissearched.Ifanamehasn’tbeenfound
aftersearchingthebuilt-inscope,thenPythonraisesaNameErrorexception,which
basicallymeansthatthenamehasn’tbeendefined(yousawthisinthepreceding
example).
Theorderinwhichthenamespacesarescannedwhenlookingforanameistherefore:
local,enclosing,global,built-in(LEGB).
Thisisallverytheoretical,solet’sseeanexample.InordertoshowyouLocaland
Enclosingnamespaces,Iwillhavetodefineafewfunctions.Don’tworryifyouarenot
familiarwiththeirsyntaxforthemoment,we’llstudyfunctionsinChapter4,Functions,
theBuildingBlocksofCode.Justrememberthatinthefollowingcode,whenyouseedef,
itmeansI’mdefiningafunction.
scopes1.py
#LocalversusGlobal
#wedefineafunction,calledlocal
deflocal():
m=7
print(m)
m=5
print(m)
#wecall,or`execute`thefunctionlocal
local()
WOW! eBook
www.wowebook.org
Intheprecedingexample,wedefinethesamenamem,bothintheglobalscopeandinthe
localone(theonedefinedbythefunctionlocal).Whenweexecutethisprogramwiththe
followingcommand(haveyouactivatedyourvirtualenv?):
$pythonscopes1.py
Weseetwonumbersprintedontheconsole:5and7.
WhathappensisthatthePythoninterpreterparsesthefile,toptobottom.First,itfindsa
coupleofcommentlines,whichareskipped,thenitparsesthedefinitionofthefunction
local.Whencalled,thisfunctiondoestwothings:itsetsupanametoanobject
representingnumber7andprintsit.ThePythoninterpreterkeepsgoinganditfinds
anothernamebinding.Thistimethebindinghappensintheglobalscopeandthevalueis
5.Thenextlineisacalltotheprintfunction,whichisexecuted(andsowegetthefirst
valueprintedontheconsole:5).
Afterthis,thereisacalltothefunctionlocal.Atthispoint,Pythonexecutesthefunction,
soatthistime,thebindingm=7happensandit’sprinted.
Oneveryimportantthingtonoticeisthatthepartofthecodethatbelongstothedefinition
ofthefunctionlocalisindentedbyfourspacesontheright.Pythoninfactdefinesscopes
byindentingthecode.Youwalkintoascopebyindentingandwalkoutofitby
unindenting.Somecodersusetwospaces,othersthree,butthesuggestednumberof
spacestouseisfour.It’sagoodmeasuretomaximizereadability.We’lltalkmoreabout
alltheconventionsyoushouldembracewhenwritingPythoncodelater.
Whatwouldhappenifweremovedthatm=7line?RemembertheLEGBrule.Python
wouldstartlookingforminthelocalscope(functionlocal),and,notfindingit,itwould
gotothenextenclosingscope.Thenextoneinthiscaseistheglobalonebecausethereis
noenclosingfunctionwrappedaroundlocal.Therefore,wewouldseetwonumber5
printedontheconsole.Let’sactuallyseehowthecodewouldlooklike:
scopes2.py
#LocalversusGlobal
deflocal():
#mdoesn'tbelongtothescopedefinedbythelocalfunction
#soPythonwillkeeplookingintothenextenclosingscope.
#misfinallyfoundintheglobalscope
print(m,'printingfromthelocalscope')
m=5
print(m,'printingfromtheglobalscope')
local()
Runningscopes2.pywillprintthis:
(.lpvenv)fab@xps:ch1$pythonscopes2.py
5printingfromtheglobalscope
5printingfromthelocalscope
WOW! eBook
www.wowebook.org
Asexpected,Pythonprintsmthefirsttime,thenwhenthefunctionlocaliscalled,misn’t
foundinitsscope,soPythonlooksforitfollowingtheLEGBchainuntilmisfoundinthe
globalscope.
Let’sseeanexamplewithanextralayer,theenclosingscope:
scopes3.py
#Local,EnclosingandGlobal
defenclosing_func():
m=13
deflocal():
#mdoesn'tbelongtothescopedefinedbythelocal
#functionsoPythonwillkeeplookingintothenext
#enclosingscope.Thistimemisfoundintheenclosing
#scope
print(m,'printingfromthelocalscope')
#callingthefunctionlocal
local()
m=5
print(m,'printingfromtheglobalscope')
enclosing_func()
Runningscopes3.pywillprintontheconsole:
(.lpvenv)fab@xps:ch1$pythonscopes3.py
5printingfromtheglobalscope
13printingfromthelocalscope
Asyoucansee,theprintinstructionfromthefunctionlocalisreferringtomasbefore.m
isstillnotdefinedwithinthefunctionitself,soPythonstartswalkingscopesfollowingthe
LEGBorder.Thistimemisfoundintheenclosingscope.
Don’tworryifthisisstillnotperfectlyclearfornow.Itwillcometoyouaswegothrough
theexamplesinthebook.TheClassessectionofthePythontutorial(official
documentation)hasaninterestingparagraphaboutscopesandnamespaces.Makesureyou
readitatsomepointifyouwishforadeeperunderstandingofthesubject.
Beforewefinishoffthischapter,Iwouldliketotalkabitmoreaboutobjects.Afterall,
basicallyeverythinginPythonisanobject,soIthinktheydeserveabitmoreattention.
Tip
Downloadingtheexamplecode
Youcandownloadtheexamplecodefilesfromyouraccountathttp://www.packtpub.com
forallthePacktPublishingbooksyouhavepurchased.Ifyoupurchasedthisbook
elsewhere,youcanvisithttp://www.packtpub.com/supportandregistertohavethefilesemaileddirectlytoyou.
WOW! eBook
www.wowebook.org
Objectandclasses
WhenIintroducedobjectsintheAproperintroductionsection,Isaidthatweusethemto
representreal-lifeobjects.Forexample,wesellgoodsofanykindontheWebnowadays
andweneedtobeabletohandle,store,andrepresentthemproperly.Butobjectsare
actuallysomuchmorethanthat.Mostofwhatyouwilleverdo,inPython,hastodowith
manipulatingobjects.
So,withoutgoingtoomuchintodetail(we’lldothatinChapter6,AdvancedConcepts–
OOP,Decorators,andIterators),Iwanttogiveyoutheinanutshellkindofexplanation
aboutclassesandobjects.
We’vealreadyseenthatobjectsarePython’sabstractionfordata.Infact,everythingin
Pythonisanobject.Numbers,strings(datastructuresthatholdtext),containers,
collections,evenfunctions.Youcanthinkofthemasiftheywereboxeswithatleastthree
features:anID(unique),atype,andavalue.
Buthowdotheycometolife?Howdowecreatethem?Howtowewriteourowncustom
objects?Theanswerliesinonesimpleword:classes.
Objectsare,infact,instancesofclasses.ThebeautyofPythonisthatclassesareobjects
themselves,butlet’snotgodownthisroad.Itleadstooneofthemostadvancedconcepts
ofthislanguage:metaclasses.We’lltalkverybrieflyabouttheminChapter6,Advanced
Concepts–OOP,Decorators,andIterators.Fornow,thebestwayforyoutogetthe
differencebetweenclassesandobjects,isbymeansofanexample.
Sayafriendtellsyou“Iboughtanewbike!”Youimmediatelyunderstandwhatshe’s
talkingabout.Haveyouseenthebike?No.Doyouknowwhatcoloritis?Nope.The
brand?Nope.Doyouknowanythingaboutit?Nope.Butatthesametime,youknow
everythingyouneedinordertounderstandwhatyourfriendmeantwhenshetoldyoushe
boughtanewbike.Youknowthatabikehastwowheelsattachedtoaframe,asaddle,
pedals,handlebars,brakes,andsoon.Inotherwords,evenifyouhaven’tseenthebike
itself,youknowtheconceptofbike.Anabstractsetoffeaturesandcharacteristicsthat
togetherformsomethingcalledbike.
Incomputerprogramming,thatiscalledaclass.It’sthatsimple.Classesareusedtocreate
objects.Infact,objectsaresaidtobeinstancesofclasses.
Inotherwords,weallknowwhatabikeis,weknowtheclass.ButthenIhavemyown
bike,whichisaninstanceoftheclassbike.Andmybikeisanobjectwithitsown
characteristicsandmethods.Youhaveyourownbike.Sameclass,butdifferentinstance.
Everybikeevercreatedintheworldisaninstanceofthebikeclass.
Let’sseeanexample.Wewillwriteaclassthatdefinesabikeandthenwe’llcreatetwo
bikes,oneredandoneblue.I’llkeepthecodeverysimple,butdon’tfretifyoudon’t
understandeverythingaboutit;allyouneedtocareaboutatthismomentistounderstand
thedifferencebetweenclassandobject(orinstanceofaclass):
bike.py
WOW! eBook
www.wowebook.org
#let'sdefinetheclassBike
classBike:
def__init__(self,colour,frame_material):
self.colour=colour
self.frame_material=frame_material
defbrake(self):
print("Braking!")
#let'screateacoupleofinstances
red_bike=Bike('Red','Carbonfiber')
blue_bike=Bike('Blue','Steel')
#let'sinspecttheobjectswehave,instancesoftheBikeclass.
print(red_bike.colour)#prints:Red
print(red_bike.frame_material)#prints:Carbonfiber
print(blue_bike.colour)#prints:Blue
print(blue_bike.frame_material)#prints:Steel
#let'sbrake!
red_bike.brake()#prints:Braking!
Tip
IhopebynowIdon’tneedtotellyoutorunthefileeverytime,right?Thefilenameis
indicatedinthefirstlineofthecodeblock.Justrun$pythonfilename,andyou’llbe
fine.
Somanyinterestingthingstonoticehere.Firstthingsfirst;thedefinitionofaclass
happenswiththeclassstatement(highlightedinthecode).Whatevercodecomesafter
theclassstatement,andisindented,iscalledthebodyoftheclass.Inourcase,thelast
linethatbelongstotheclassdefinitionistheprint("Braking!")one.
Afterhavingdefinedtheclasswe’rereadytocreateinstances.Youcanseethattheclass
bodyhoststhedefinitionoftwomethods.Amethodisbasically(andsimplistically)a
functionthatbelongstoaclass.
Thefirstmethod,__init__isaninitializer.ItusessomePythonmagictosetupthe
objectswiththevalueswepasswhenwecreateit.
Note
Everymethodthathasleadingandtrailingdoubleunderscore,inPython,iscalledmagic
method.MagicmethodsareusedbyPythonforamultitudeofdifferentpurposes,hence
it’sneveragoodideatonameacustommethodusingtwoleadingandtrailing
underscores.ThisnamingconventionisbestlefttoPython.
Theothermethodwedefined,brake,isjustanexampleofanadditionalmethodthatwe
couldcallifwewantedtobrakethebike.Itcontainsjustaprintstatement,ofcourse,it’s
anexample.
Wecreatedtwobikesthen.Onehasredcolorandacarbonfiberframe,andtheotherone
hasbluecolorandsteelframe.Wepassthosevaluesuponcreation.Aftercreation,we
printoutthecolorpropertyandframetypeoftheredbike,andtheframetypeoftheblue
WOW! eBook
www.wowebook.org
onejustasanexample.Wealsocallthebrakemethodofthered_bike.
Onelastthingtonotice.YourememberItoldyouthatthesetofattributesofanobjectis
consideredtobeanamespace?Ihopeit’sclearernow,whatImeant.Youseethatby
gettingtotheframe_typepropertythroughdifferentnamespaces(red_bike,blue_bike)
weobtaindifferentvalues.Nooverlapping,noconfusion.
Thedot(.)operatorisofcoursethemeansweusetowalkintoanamespace,inthecaseof
objectsaswell.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Guidelinesonhowtowritegoodcode
Writinggoodcodeisnotaseasyasitseems.AsIalreadysaidbefore,goodcodeexposes
alonglistofqualitiesthatisquitehardtoputtogether.Writinggoodcodeis,tosome
extent,anart.Regardlessofwhereonthepathyouwillbehappytosettle,thereis
somethingthatyoucanembracewhichwillmakeyourcodeinstantlybetter:PEP8.
AccordingtoWikipedia:
“Python’sdevelopmentisconductedlargelythroughthePythonEnhancement
Proposal(PEP)process.ThePEPprocessistheprimarymechanismforproposing
majornewfeatures,forcollectingcommunityinputonanissue,andfordocumenting
thedesigndecisionsthathavegoneintoPython.”
AmongallthePEPs,probablythemostfamousoneisPEP8.Itlaysoutasimplebut
effectivesetofguidelinestodefinePythonaestheticsothatwewritebeautifulPython
code.Ifyoutakeonesuggestionoutofthischapter,pleaseletitbethis:useit.Embraceit.
Youwillthankmelater.
Codingtodayisnolongeracheck-in/check-outbusiness.Rather,it’smoreofasocial
effort.Severaldeveloperscollaboratetoapieceofcodethroughtoolslikegitand
mercurial,andtheresultiscodethatisfatheredbymanydifferenthands.
Note
GitandMercurialareprobablythemostuseddistributedrevisioncontrolsystemstoday.
Theyareessentialtoolsdesignedtohelpteamsofdeveloperscollaborateonthesame
software.
Thesedays,morethanever,weneedtohaveaconsistentwayofwritingcode,sothat
readabilityismaximized.WhenalldevelopersofacompanyabidewithPEP8,it’snot
uncommonforanyofthemlandingonapieceofcodetothinktheywroteitthemselves.It
actuallyhappenstomeallthetime(IalwaysforgetthecodeIwrite).
Thishasatremendousadvantage:whenyoureadcodethatyoucouldhavewritten
yourself,youreaditeasily.Withoutaconvention,everycoderwouldstructurethecode
thewaytheylikemost,orsimplythewaytheyweretaughtorareusedto,andthiswould
meanhavingtointerpreteverylineaccordingtosomeoneelse’sstyle.Itwouldmean
havingtolosemuchmoretimejusttryingtounderstandit.ThankstoPEP8,wecanavoid
this.I’msuchafanofitthatIwon’tsignoffacodereviewifthecodedoesn’trespectit.
Sopleasetakethetimetostudyit,it’sveryimportant.
Intheexamplesofthisbook,IwilltrytorespectitasmuchasIcan.Unfortunately,Idon’t
havetheluxuryof79characters(whichisthemaximumlinelengthsuggestedbyPEP*),
andIwillhavetocutdownonblanklinesandotherthings,butIpromiseyouI’lltryto
layoutmycodesothatit’sasreadableaspossible.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
ThePythonculture
Pythonhasbeenadoptedwidelyinallcodingindustries.It’susedbymanydifferent
companiesformanydifferentpurposes,andit’salsousedineducation(it’sanexcellent
languageforthatpurpose,becauseofitsmanyqualitiesandthefactthatit’seasytolearn).
OneofthereasonsPythonissopopulartodayisthatthecommunityarounditisvast,
vibrant,andfullofbrilliantpeople.Manyeventsareorganizedallovertheworld,mostly
eitheraroundPythonoritsmainwebframework,Django.
Pythonisopen,andveryoftensoarethemindsofthosewhoembraceit.Checkoutthe
communitypageonthePythonwebsiteformoreinformationandgetinvolved!
ThereisanotheraspecttoPythonwhichrevolvesaroundthenotionofbeingPythonic.It
hastodowiththefactthatPythonallowsyoutousesomeidiomsthataren’tfound
elsewhere,atleastnotinthesameformoreasinessofuse(Ifeelquiteclaustrophobic
whenIhavetocodeinalanguagewhichisnotPythonnow).
Anyway,overtheyears,thisconceptofbeingPythonichasemergedand,thewayI
understandit,issomethingalongthelinesofdoingthingsthewaytheyaresupposedtobe
doneinPython.
TohelpyouunderstandalittlebitmoreaboutPython’scultureandaboutbeingPythonic,I
willshowyoutheZenofPython.AlovelyEastereggthatisverypopular.Openupa
Pythonconsoleandtypeimportthis.Whatfollowsistheresultofthisline:
>>>importthis
TheZenofPython,byTimPeters
Beautifulisbetterthanugly.
Explicitisbetterthanimplicit.
Simpleisbetterthancomplex.
Complexisbetterthancomplicated.
Flatisbetterthannested.
Sparseisbetterthandense.
Readabilitycounts.
Specialcasesaren'tspecialenoughtobreaktherules.
Althoughpracticalitybeatspurity.
Errorsshouldneverpasssilently.
Unlessexplicitlysilenced.
Inthefaceofambiguity,refusethetemptationtoguess.
Thereshouldbeone--andpreferablyonlyone--obviouswaytodoit.
Althoughthatwaymaynotbeobviousatfirstunlessyou'reDutch.
Nowisbetterthannever.
Althoughneverisoftenbetterthan*right*now.
Iftheimplementationishardtoexplain,it'sabadidea.
Iftheimplementationiseasytoexplain,itmaybeagoodidea.
Namespacesareonehonkinggreatidea—let'sdomoreofthose!
Therearetwolevelsofreadinghere.Oneistoconsideritasasetofguidelinesthathave
beenputdowninafunway.Theotheroneistokeepitinmind,andmaybereaditoncein
awhile,tryingtounderstandhowitreferstosomethingdeeper.SomePython
WOW! eBook
www.wowebook.org
characteristicsthatyouwillhavetounderstanddeeplyinordertowritePythontheway
it’ssupposedtobewritten.Startwiththefunlevel,andthendigdeeper.Alwaysdig
deeper.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
AnoteontheIDEs
JustafewwordsaboutIntegratedDevelopmentEnvironments(IDEs).Tofollowthe
examplesinthisbookyoudon’tneedone,anytexteditorwilldofine.Ifyouwanttohave
moreadvancedfeaturessuchassyntaxcoloringandautocompletion,youwillhaveto
fetchyourselfanIDE.YoucanfindacomprehensivelistofopensourceIDEs(just
Google“pythonides”)onthePythonwebsite.IpersonallyuseSublimeTexteditor.It’s
freetotryoutanditcostsjustafewdollars.IhavetriedmanyIDEsinmylife,butthisis
theonethatmakesmemostproductive.
Twoextremelyimportantpiecesofadvice:
WhateverIDEyouwillchosetouse,trytolearnitwellsothatyoucanexploitits
strengths,butdon’tdependonit.ExerciseyourselftoworkwithVIM(oranyother
texteditor)onceinawhile,learntobeabletodosomeworkonanyplatform,with
anysetoftools.
Whatevertexteditor/IDEyouwilluse,whenitcomestowritingPython,indentation
isfourspaces.Don’tusetabs,don’tmixthemwithspaces.Usefourspaces,nottwo,
notthree,notfive.Justusefour.Thewholeworldworkslikethat,andyoudon’t
wanttobecomeanoutcastbecauseyouwerefondofthethree-spacelayout.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,westartedtoexploretheworldofprogrammingandthatofPython.We’ve
barelyscratchedthesurface,justalittle,touchingconceptsthatwillbediscussedlateron
inthebookingreaterdetail.
WetalkedaboutPython’smainfeatures,whoisusingitandforwhat,andwhatarethe
differentwaysinwhichwecanwriteaPythonprogram.
Inthelastpartofthechapter,weflewoverthefundamentalnotionsofnamespace,scope,
class,andobject.WealsosawhowPythoncodecanbeorganizedusingmodulesand
packages.
Onapracticallevel,welearnedhowtoinstallPythononoursystem,howtomakesurewe
havethetoolsweneed,pipandvirtualenv,andwealsocreatedandactivatedourfirst
virtualenvironment.Thiswillallowustoworkinaself-containedenvironmentwithout
theriskofcompromisingthePythonsysteminstallation.
Nowyou’rereadytostartthisjourneywithme.Allyouneedisenthusiasm,anactivated
virtualenvironment,thisbook,yourfingers,andsomecoffee.
Trytofollowtheexamples,I’llkeepthemsimpleandshort.Ifyouputthemunderyour
fingertips,youwillretainthemmuchbetterthanifyoujustreadthem.
Inthenextchapter,wewillexplorePython’srichsetofbuilt-indatatypes.There’smuch
tocoverandmuchtolearn!
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter2.Built-inDataTypes
“Data!Data!Data!”hecriedimpatiently.“Ican’tmakebrickswithoutclay.” —SherlockHolmes-TheAdventureoftheCopperBeeches
Everythingyoudowithacomputerismanagingdata.Datacomesinmanydifferent
shapesandflavors.It’sthemusicyoulisten,themovieyoustream,thePDFsyouopen.
Eventhechapteryou’rereadingatthisverymomentisjustafile,whichisdata.
Datacanbesimple,anintegernumbertorepresentanage,orcomplex,likeanorder
placedonawebsite.Itcanbeaboutasingleobjectoraboutacollectionofthem.
Datacanevenbeaboutdata,thatis,metadata.Datathatdescribesthedesignofotherdata
structuresordatathatdescribesapplicationdataoritscontext.
InPython,objectsareabstractionfordata,andPythonhasanamazingvarietyofdata
structuresthatyoucanusetorepresentdata,orcombinethemtocreateyourowncustom
data.Beforewedelveintothespecifics,Iwantyoutobeveryclearaboutobjectsin
Python,solet’stalkalittlebitmoreaboutthem.
WOW! eBook
www.wowebook.org
Everythingisanobject
Aswealreadysaid,everythinginPythonisanobject.Butwhatreallyhappenswhenyou
typeaninstructionlikeage=42inaPythonmodule?
Tip
Ifyougotohttp://pythontutor.com/,youcantypethatinstructionintoatextboxandget
itsvisualrepresentation.Keepthiswebsiteinmind,it’sveryusefultoconsolidateyour
understandingofwhatgoesonbehindthescenes.
So,whathappensisthatanobjectiscreated.Itgetsanid,thetypeissettoint(integer
number),andthevalueto42.Anameageisplacedintheglobalnamespace,pointingto
thatobject.Therefore,wheneverweareintheglobalnamespace,aftertheexecutionof
thatline,wecanretrievethatobjectbysimplyaccessingitthroughitsname:age.
Ifyouweretomovehouse,youwouldputalltheknives,forks,andspoonsinaboxand
labelitcutlery.Canyouseeit’sexactlythesameconcept?Here’sascreenshotofhowit
maylooklike(youmayhavetotweakthesettingstogettothesameview):
So,fortherestofthischapter,wheneveryoureadsomethingsuchasname=some_value,
thinkofanameplacedinthenamespacethatistiedtothescopeinwhichtheinstruction
waswritten,withanicearrowpointingtoanobjectthathasanid,atype,andavalue.
Thereisalittlebitmoretosayaboutthismechanism,butit’smucheasiertotalkaboutit
overanexample,sowe’llgetbacktothislater.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Mutableorimmutable?Thatisthe
question
AfirstfundamentaldistinctionthatPythonmakesondataisaboutwhetherornotthe
valueofanobjectchanges.Ifthevaluecanchange,theobjectiscalledmutable,whileif
thevaluecannotchange,theobjectiscalledimmutable.
Itisveryimportantthatyouunderstandthedistinctionbetweenmutableandimmutable
becauseitaffectsthecodeyouwrite,sohere’saquestion:
>>>age=42
>>>age
42
>>>age=43#A
>>>age
43
Intheprecedingcode,ontheline#A,haveIchangedthevalueofage?Well,no.Butnow
it’s43(Ihearyousay…).Yes,it’s43,but42wasanintegernumber,ofthetypeint,
whichisimmutable.So,whathappenedisreallythatonthefirstline,ageisanamethatis
settopointtoanintobject,whosevalueis42.Whenwetypeage=43,whathappensis
thatanotherobjectiscreated,ofthetypeintandvalue43(also,theidwillbedifferent),
andthenameageissettopointtoit.So,wedidn’tchangethat42to43.Weactuallyjust
pointedagetoadifferentlocation:thenewintobjectwhosevalueis43.Let’sseethe
samecodealsoprintingtheIDs:
>>>age=42
>>>id(age)
10456352
>>>age=43
>>>id(age)
10456384
NoticethatweprinttheIDsbycallingthebuilt-inidfunction.Asyoucansee,theyare
different,asexpected.Bearinmindthatagepointstooneobjectatatime:42first,then
43.Nevertogether.
Now,let’sseethesameexampleusingamutableobject.Forthisexample,let’sjustusea
Personobject,thathasapropertyage:
>>>fab=Person(age=39)
>>>fab.age
39
>>>id(fab)
139632387887456
>>>fab.age=29#Iwish!
>>>id(fab)
139632387887456#stillthesameid
Inthiscase,IsetupanobjectfabwhosetypeisPerson(acustomclass).Oncreation,the
objectisgiventheageof39.I’mprintingit,alongwiththeobjectid,rightafterwards.
WOW! eBook
www.wowebook.org
Noticethat,evenafterIchangeagetobe29,theIDoffabstaysthesame(whiletheIDof
agehaschanged,ofcourse).CustomobjectsinPythonaremutable(unlessyoucodethem
nottobe).Keepthisconceptinmind,it’sveryimportant.I’llremindyouaboutitthrough
therestofthechapter.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Numbers
Let’sstartbyexploringPython’sbuilt-indatatypesfornumbers.Pythonwasdesignedby
amanwithamaster’sdegreeinmathematicsandcomputerscience,soit’sonlylogical
thatithasamazingsupportfornumbers.
Numbersareimmutableobjects.
WOW! eBook
www.wowebook.org
Integers
Pythonintegershaveunlimitedrange,subjectonlytotheavailablevirtualmemory.This
meansthatitdoesn’treallymatterhowbiganumberyouwanttostore:aslongasitcanfit
inyourcomputer’smemory,Pythonwilltakecareofit.Integernumberscanbepositive,
negative,and0(zero).Theysupportallthebasicmathematicaloperations,asshowninthe
followingexample:
>>>a=12
>>>b=3
>>>a+b#addition
15
>>>b-a#subtraction
-9
>>>a//b#integerdivision
4
>>>a/b#truedivision
4.0
>>>a*b#multiplication
36
>>>b**a#poweroperator
531441
>>>2**1024#averybignumber,Pythonhandlesitgracefully
17976931348623159077293051907890247336179769789423065727343008115
77326758055009631327084773224075360211201138798713933576587897688
14416622492847430639474124377767893424865485276302219601246094119
45308295208500576883815068234246288147391311054082723716335051068
4586298239947245938479716304835356329624224137216
Theprecedingcodeshouldbeeasytounderstand.Justnoticeoneimportantthing:Python
hastwodivisionoperators,oneperformstheso-calledtruedivision(/),whichreturnsthe
quotientoftheoperands,andtheotherone,theso-calledintegerdivision(//),which
returnstheflooredquotientoftheoperands.Seehowthatisdifferentforpositiveand
negativenumbers:
>>>7/4#truedivision
1.75
>>>7//4#integerdivision,flooringreturns1
1
>>>-7/4#truedivisionagain,resultisoppositeofprevious
-1.75
>>>-7//4#integerdiv.,resultnottheoppositeofprevious
-2
Thisisaninterestingexample.Ifyouwereexpectinga-1onthelastline,don’tfeelbad,
it’sjustthewayPythonworks.TheresultofanintegerdivisioninPythonisalways
roundedtowardsminusinfinity.Ifinsteadofflooringyouwanttotruncateanumbertoan
integer,youcanusethebuilt-inintfunction,likeshowninthefollowingexample:
>>>int(1.75)
1
>>>int(-1.75)
-1
WOW! eBook
www.wowebook.org
Noticethattruncationisdonetowards0.
Thereisalsoanoperatortocalculatetheremainderofadivision.It’scalledmodulo
operator,andit’srepresentedbyapercent(%):
>>>10%3#remainderofthedivision10//3
1
>>>10%4#remainderofthedivision10//4
2
WOW! eBook
www.wowebook.org
Booleans
Booleanalgebraisthatsubsetofalgebrainwhichthevaluesofthevariablesarethetruth
values:trueandfalse.InPython,TrueandFalsearetwokeywordsthatareusedto
representtruthvalues.Booleansareasubclassofintegers,andbehaverespectivelylike1
and0.TheequivalentoftheintclassforBooleansistheboolclass,whichreturnseither
TrueorFalse.Everybuilt-inPythonobjecthasavalueintheBooleancontext,which
meanstheybasicallyevaluatetoeitherTrueorFalsewhenfedtotheboolfunction.We’ll
seeallaboutthisinChapter3,IteratingandMakingDecisions.
BooleanvaluescanbecombinedinBooleanexpressionsusingthelogicaloperatorsand,
or,andnot.Again,we’llseetheminfullinthenextchapter,sofornowlet’sjustseea
simpleexample:
>>>int(True)#Truebehaveslike1
1
>>>int(False)#Falsebehaveslike0
0
>>>bool(1)#1evaluatestoTrueinabooleancontext
True
>>>bool(-42)#andsodoeseverynon-zeronumber
True
>>>bool(0)#0evaluatestoFalse
False
>>>#quickpeakattheoperators(and,or,not)
>>>notTrue
False
>>>notFalse
True
>>>TrueandTrue
True
>>>FalseorTrue
True
YoucanseethatTrueandFalsearesubclassesofintegerswhenyoutrytoaddthem.
Pythonupcaststhemtointegersandperformsaddition:
>>>1+True
2
>>>False+42
42
>>>7-True
6
Note
Upcastingisatypeconversionoperationthatgoesfromasubclasstoitsparent.Inthe
examplepresentedhere,TrueandFalse,whichbelongtoaclassderivedfromtheinteger
class,areconvertedbacktointegerswhenneeded.Thistopicisaboutinheritanceandwill
beexplainedindetailinChapter6,AdvancedConcepts–OOP,Decorators,andIterators.
WOW! eBook
www.wowebook.org
Reals
Realnumbers,orfloatingpointnumbers,arerepresentedinPythonaccordingtotheIEEE
754double-precisionbinaryfloating-pointformat,whichisstoredin64bitsof
informationdividedintothreesections:sign,exponent,andmantissa.
Note
QuenchyourthirstforknowledgeaboutthisformatonWikipedia:
http://en.wikipedia.org/wiki/Double-precision_floating-point_format
Usuallyprogramminglanguagesgivecoderstwodifferentformats:singleanddouble
precision.Theformertakingup32bitsofmemory,andthelatter64.Pythonsupportsonly
thedoubleformat.Let’sseeasimpleexample:
>>>pi=3.1415926536#howmanydigitsofPIcanyouremember?
>>>radius=4.5
>>>area=pi*(radius**2)
>>>area
63.61725123519331
Note
Inthecalculationofthearea,Iwrappedtheradius**2withinbraces.Eventhoughthat
wasn’tnecessarybecausethepoweroperatorhashigherprecedencethanthe
multiplicationone,Ithinktheformulareadsmoreeasilylikethat.
Thesys.float_infostructsequenceholdsinformationabouthowfloatingpointnumbers
willbehaveonyoursystem.ThisiswhatIseeonmybox:
>>>importsys
>>>sys.float_info
sys.float_info(max=1.7976931348623157e+308,max_exp=1024,max_10_exp=308,
min=2.2250738585072014e-308,min_exp=-1021,min_10_exp=-307,dig=15,
mant_dig=53,epsilon=2.220446049250313e-16,radix=2,rounds=1)
Let’smakeafewconsiderationshere:wehave64bitstorepresentfloatnumbers.This
meanswecanrepresentatmost2**64==18,446,744,073,709,551,616numbers
withthatamountofbits.Takealookatthemaxandepsilonvalueforthefloatnumbers,
andyou’llrealizeit’simpossibletorepresentthemall.Thereisjustnotenoughspaceso
theyareapproximatedtotheclosestrepresentablenumber.Youprobablythinkthatonly
extremelybigorextremelysmallnumberssufferfromthisissue.Well,thinkagain:
>>>3*0.1–0.3#thisshouldbe0!!!
5.551115123125783e-17
Whatdoesthistellyou?Ittellsyouthatdoubleprecisionnumberssufferfrom
approximationissuesevenwhenitcomestosimplenumberslike0.1or0.3.Whyisthis
important?Itcanbeabigproblemifyou’rehandlingprices,orfinancialcalculations,or
anykindofdatathatneedsnottobeapproximated.Don’tworry,Pythongivesyouthe
Decimaltype,whichdoesn’tsufferfromtheseissues,we’llseetheminabit.
WOW! eBook
www.wowebook.org
Complexnumbers
Pythongivesyoucomplexnumberssupportoutofthebox.Ifyoudon’tknowwhat
complexnumbersare,youcanlookthemupontheWeb.Theyarenumbersthatcanbe
expressedintheforma+ibwhereaandbarerealnumbers,andi(orjifyou’rean
engineer)istheimaginaryunit,thatis,thesquarerootof-1.aandbarecalled
respectivelytherealandimaginarypartofthenumber.
It’sactuallyunlikelyyou’llbeusingthem,unlessyou’recodingsomethingscientific.
Let’sseeasmallexample:
>>>c=3.14+2.73j
>>>c.real#realpart
3.14
>>>c.imag#imaginarypart
2.73
>>>c.conjugate()#conjugateofA+BjisA-Bj
(3.14-2.73j)
>>>c*2#multiplicationisallowed
(6.28+5.46j)
>>>c**2#poweroperationaswell
(2.4067000000000007+17.1444j)
>>>d=1+1j#additionandsubtractionaswell
>>>c-d
(2.14+1.73j)
WOW! eBook
www.wowebook.org
Fractionsanddecimals
Let’sfinishthetourofthenumberdepartmentwithalookatfractionsanddecimals.
Fractionsholdarationalnumeratoranddenominatorintheirlowestforms.Let’sseea
quickexample:
>>>fromfractionsimportFraction
>>>Fraction(10,6)#madhatter?
Fraction(5,3)#noticeit'sbeenreducedtolowestterms
>>>Fraction(1,3)+Fraction(2,3)#1/3+2/3=3/3=1/1
Fraction(1,1)
>>>f=Fraction(10,6)
>>>f.numerator
5
>>>f.denominator
3
Althoughtheycanbeveryusefulattimes,it’snotthatcommontospotthemin
commercialsoftware.Mucheasierinstead,istoseedecimalnumbersbeingusedinall
thosecontextswhereprecisioniseverything,forexample,scientificandfinancial
calculations.
Note
It’simportanttorememberthatarbitraryprecisiondecimalnumberscomeatapricein
performance,ofcourse.Theamountofdatatobestoredforeachnumberisfargreater
thanitisforfractionsorfloatsaswellasthewaytheyarehandled,whichrequiresthe
Pythoninterpretermuchmoreworkbehindthescenes.Anotherinterestingthingtoknow
isthatyoucangetandsettheprecisionbyaccessingdecimal.getcontext().prec.
Let’sseeaquickexamplewithDecimalnumbers:
>>>fromdecimalimportDecimalasD#renameforbrevity
>>>D(3.14)#pi,fromfloat,soapproximationissues
Decimal('3.140000000000000124344978758017532527446746826171875')
>>>D('3.14')#pi,fromastring,sonoapproximationissues
Decimal('3.14')
>>>D(0.1)*D(3)-D(0.3)#fromfloat,westillhavetheissue
Decimal('2.775557561565156540423631668E-17')
>>>D('0.1')*D(3)-D('0.3')#fromstring,allperfect
Decimal('0.0')
NoticethatwhenweconstructaDecimalnumberfromafloat,ittakesonallthe
approximationissuesthefloatmaycomefrom.Ontheotherhand,whentheDecimalhas
noapproximationissues,forexample,whenwefeedanintorastringrepresentationto
theconstructor,thenthecalculationhasnoquirkybehavior.Whenitcomestomoney,use
decimals.
Thisconcludesourintroductiontobuilt-innumerictypes,let’snowseesequences.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Immutablesequences
Let’sstartwithimmutablesequences:strings,tuples,andbytes.
WOW! eBook
www.wowebook.org
Stringsandbytes
TextualdatainPythonishandledwithstrobjects,morecommonlyknownasstrings.
Theyareimmutablesequencesofunicodecodepoints.Unicodecodepointscanrepresent
acharacter,butcanalsohaveothermeanings,suchasformattingdataforexample.
Python,unlikeotherlanguages,doesn’thaveachartype,soasinglecharacterisrendered
simplybyastringoflength1.Unicodeisanexcellentwaytohandledata,andshouldbe
usedfortheinternalsofanyapplication.Whenitcomestostoretextualdatathough,or
senditonthenetwork,youmaywanttoencodeit,usinganappropriateencodingforthe
mediumyou’reusing.StringliteralsarewritteninPythonusingsingle,doubleortriple
quotes(bothsingleordouble).Ifbuiltwithtriplequotes,astringcanspanonmultiple
lines.Anexamplewillclarifythepicture:
>>>#4waystomakeastring
>>>str1='Thisisastring.Webuiltitwithsinglequotes.'
>>>str2="Thisisalsoastring,butbuiltwithdoublequotes."
>>>str3='''Thisisbuiltusingtriplequotes,
...soitcanspanmultiplelines.'''
>>>str4="""Thistoo
...isamultilineone
...builtwithtripledouble-quotes."""
>>>str4#A
'Thistoo\nisamultilineone\nbuiltwithtripledouble-quotes.'
>>>print(str4)#B
Thistoo
isamultilineone
builtwithtripledouble-quotes.
In#Aand#B,weprintstr4,firstimplicitly,thenexplicitlyusingtheprintfunction.A
niceexercisewouldbetofindoutwhytheyaredifferent.Areyouuptothechallenge?
(hint,lookupthestrfunction)
Strings,likeanysequence,havealength.Youcangetthisbycallingthelenfunction:
>>>len(str1)
49
Encodinganddecodingstrings
Usingtheencode/decodemethods,wecanencodeunicodestringsanddecodebytes
objects.Utf-8isavariablelengthcharacterencoding,capableofencodingallpossible
unicodecodepoints.ItisthedominantencodingfortheWeb(andnotonly).Noticealso
thatbyaddingaliteralbinfrontofastringdeclaration,we’recreatingabytesobject.
>>>s="Thisisüŋíc0de"#unicodestring:codepoints
>>>type(s)
<class'str'>
>>>encoded_s=s.encode('utf-8')#utf-8encodedversionofs
>>>encoded_s
b'Thisis\xc3\xbc\xc5\x8b\xc3\xadc0de'#result:bytesobject
>>>type(encoded_s)#anotherwaytoverifyit
<class'bytes'>
>>>encoded_s.decode('utf-8')#let'sreverttotheoriginal
WOW! eBook
www.wowebook.org
'Thisisüŋíc0de'
>>>bytes_obj=b"Abytesobject"#abytesobject
>>>type(bytes_obj)
<class'bytes'>
Indexingandslicingstrings
Whenmanipulatingsequences,it’sverycommontohavetoaccessthematoneprecise
position(indexing),ortogetasubsequenceoutofthem(slicing).Whendealingwith
immutablesequences,bothoperationsareread-only.
Whileindexingcomesinoneform,azero-basedaccesstoanypositionwithinthe
sequence,slicingcomesindifferentforms.Whenyougetasliceofasequence,youcan
specifythestartandstoppositions,andthestep.Theyareseparatedwithacolon(:)
likethis:my_sequence[start:stop:step].Alltheargumentsareoptional,startis
inclusive,stopisexclusive.It’smucheasiertoshowanexample,ratherthanexplainthem
furtherinwords:
>>>s="Thetroubleisyouthinkyouhavetime."
>>>s[0]#indexingatposition0,whichisthefirstchar
'T'
>>>s[5]#indexingatposition5,whichisthesixthchar
'r'
>>>s[:4]#slicing,wespecifyonlythestopposition
'The'
>>>s[4:]#slicing,wespecifyonlythestartposition
'troubleisyouthinkyouhavetime.'
>>>s[2:14]#slicing,bothstartandstoppositions
'etroubleis'
>>>s[2:14:3]#slicing,start,stopandstep(every3chars)
'erb'
>>>s[:]#quickwayofmakingacopy
'Thetroubleisyouthinkyouhavetime.'
Ofallthelines,thelastoneisprobablythemostinteresting.Ifyoudon’tspecifya
parameter,Pythonwillfillinthedefaultforyou.Inthiscase,startwillbethestartofthe
string,stopwillbetheendofthesting,andstepwillbethedefault1.Thisisaneasyand
quickwayofobtainingacopyofthestrings(samevalue,butdifferentobject).Canyou
findawaytogetthereversedcopyofastringusingslicing?(don’tlookitup,finditfor
yourself)
WOW! eBook
www.wowebook.org
Tuples
Thelastimmutablesequencetypewe’regoingtoseeisthetuple.Atupleisasequenceof
arbitraryPythonobjects.Inatuple,itemsareseparatedbycommas.Theyareused
everywhereinPython,becausetheyallowforpatternsthatarehardtoreproduceinother
languages.Sometimestuplesareusedimplicitly,forexampletosetupmultiplevariables
ononeline,ortoallowafunctiontoreturnmultipledifferentobjects(usuallyafunction
returnsoneobjectonly,inmanyotherlanguages),andeveninthePythonconsole,you
canusetuplesimplicitlytoprintmultipleelementswithonesingleinstruction.We’llsee
examplesforallthesecases:
>>>t=()#emptytuple
>>>type(t)
<class'tuple'>
>>>one_element_tuple=(42,)#youneedthecomma!
>>>three_elements_tuple=(1,3,5)
>>>a,b,c=1,2,3#tupleformultipleassignment
>>>a,b,c#implicittupletoprintwithoneinstruction
(1,2,3)
>>>3inthree_elements_tuple#membershiptest
True
Noticethatthemembershipoperatorincanalsobeusedwithlists,strings,dictionaries,
andingeneralwithcollectionandsequenceobjects.
Note
Noticethattocreateatuplewithoneitem,weneedtoputthatcommaaftertheitem.The
reasonisthatwithoutthecommathatitemisjustitselfwrappedinbraces,kindofina
redundantmathematicalexpression.Noticealsothatonassignment,bracesareoptionalso
my_tuple=1,2,3isthesameasmy_tuple=(1,2,3).
Onethingthattupleassignmentallowsustodo,isone-lineswaps,withnoneedforathird
temporaryvariable.Let’sseefirstamoretraditionalwayofdoingit:
>>>a,b=1,2
>>>c=a#weneedthreelinesandatemporaryvarc
>>>a=b
>>>b=c
>>>a,b#aandbhavebeenswapped
(2,1)
Andnowlet’sseehowwewoulddoitinPython:
>>>a,b=b,a#thisisthePythonicwaytodoit
>>>a,b
(1,2)
TakealookatthelinethatshowsyouthePythonicwayofswappingtwovalues:doyou
rememberwhatIwroteinChapter1,IntroductionandFirstSteps–TakeaDeepBreath.
APythonprogramistypicallyone-fifthtoone-thirdthesizeofequivalentJavaorC++
code,andfeatureslikeone-lineswapscontributetothis.Pythoniselegant,whereelegance
WOW! eBook
www.wowebook.org
inthiscontextmeansalsoeconomy.
Becausetheyareimmutable,tuplescanbeusedaskeysfordictionaries(we’llseethis
shortly).Thedictobjectsneedkeystobeimmutablebecauseiftheycouldchange,then
thevaluetheyreferencewouldn’tbefoundanymore(becausethepathtoitdependson
thekey).Ifyouareintodatastructures,youknowhowniceafeaturethisoneistohave.
Tome,tuplesarePython’sbuilt-indatathatmostcloselyrepresentamathematicalvector.
Thisdoesn’tmeanthatthiswasthereasonforwhichtheywerecreatedthough.Tuples
usuallycontainanheterogeneoussequenceofelements,whileontheotherhandlistsare
mostofthetimeshomogeneous.Moreover,tuplesarenormallyaccessedviaunpackingor
indexing,whilelistsareusuallyiteratedover.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Mutablesequences
Mutablesequencesdifferfromtheirimmutablesistersinthattheycanbechangedafter
creation.TherearetwomutablesequencetypesinPython:listsandbytearrays.Isaid
beforethatthedictionaryisthekingofdatastructuresinPython.Iguessthismakesthe
listitsrightfulqueen.
WOW! eBook
www.wowebook.org
Lists
Pythonlistsaremutablesequences.Theyareverysimilartotuples,buttheydon’thave
therestrictionsduetoimmutability.Listsarecommonlyusedtostorecollectionsof
homogeneousobjects,butthereisnothingpreventingyoutostoreheterogeneous
collectionsaswell.Listscanbecreatedinmanydifferentways,let’sseeanexample:
>>>[]#emptylist
[]
>>>list()#sameas[]
[]
>>>[1,2,3]#aswithtuples,itemsarecommaseparated
[1,2,3]
>>>[x+5forxin[2,3,4]]#Pythonismagic
[7,8,9]
>>>list((1,3,5,7,9))#listfromatuple
[1,3,5,7,9]
>>>list('hello')#listfromastring
['h','e','l','l','o']
Inthepreviousexample,Ishowedyouhowtocreatealistusingdifferenttechniques.I
wouldlikeyoutotakeagoodlookatthelinethatsaysPythonismagic,whichIamnot
expectingyoutofullyunderstandatthispoint(unlessyoucheatedandyou’renota
novice!).Thatiscalledalistcomprehension,averypowerfulfunctionalfeatureof
Python,whichwe’llseeindetailinChapter5,SavingTimeandMemory.Ijustwantedto
makeyourmouthwateratthispoint.
Creatinglistsisgood,buttherealfuncomeswhenweusethem,solet’sseethemain
methodstheygiftuswith:
>>>a=[1,2,1,3]
>>>a.append(13)#wecanappendanythingattheend
>>>a
[1,2,1,3,13]
>>>a.count(1)#howmany`1`arethereinthelist?
2
>>>a.extend([5,7])#extendthelistbyanother(orsequence)
>>>a
[1,2,1,3,13,5,7]
>>>a.index(13)#positionof`13`inthelist(0-basedindexing)
4
>>>a.insert(0,17)#insert`17`atposition0
>>>a
[17,1,2,1,3,13,5,7]
>>>a.pop()#pop(removeandreturn)lastelement
7
>>>a.pop(3)#popelementatposition3
1
>>>a
[17,1,2,3,13,5]
>>>a.remove(17)#remove`17`fromthelist
>>>a
[1,2,3,13,5]
WOW! eBook
www.wowebook.org
>>>a.reverse()#reversetheorderoftheelementsinthelist
>>>a
[5,13,3,2,1]
>>>a.sort()#sortthelist
>>>a
[1,2,3,5,13]
>>>a.clear()#removeallelementsfromthelist
>>>a
[]
Theprecedingcodegivesyouaroundupoflist’smainmethods.Iwanttoshowyouhow
powerfultheyare,usingextendasanexample.Youcanextendlistsusinganysequence
type:
>>>a=list('hello')#makesalistfromastring
>>>a
['h','e','l','l','o']
>>>a.append(100)#append100,heterogeneoustype
>>>a
['h','e','l','l','o',100]
>>>a.extend((1,2,3))#extendusingtuple
>>>a
['h','e','l','l','o',100,1,2,3]
>>>a.extend('...')#extendusingstring
>>>a
['h','e','l','l','o',100,1,2,3,'.','.','.']
Now,let’sseewhatarethemostcommonoperationsyoucandowithlists:
>>>a=[1,3,5,7]
>>>min(a)#minimumvalueinthelist
1
>>>max(a)#maximumvalueinthelist
7
>>>sum(a)#sumofallvaluesinthelist
16
>>>len(a)#numberofelementsinthelist
4
>>>b=[6,7,8]
>>>a+b#`+`withlistmeansconcatenation
[1,3,5,7,6,7,8]
>>>a*2#`*`hasalsoaspecialmeaning
[1,3,5,7,1,3,5,7]
Thelasttwolinesintheprecedingcodearequiteinterestingbecausetheyintroduceusto
aconceptcalledoperatoroverloading.Inshort,itmeansthatoperatorssuchas+,-.*,%,
andsoon,mayrepresentdifferentoperationsaccordingtothecontexttheyareusedin.It
doesn’tmakeanysensetosumtwolists,right?Therefore,the+signisusedto
concatenatethem.Hence,the*signisusedtoconcatenatethelisttoitselfaccordingtothe
rightoperand.Now,let’stakeastepfurtherdowntherabbitholeandseesomethingalittle
moreinteresting.Iwanttoshowyouhowpowerfulthesortmethodcanbeandhoweasyit
isinPythontoachieveresultsthatrequireagreatdealofeffortinotherlanguages:
>>>fromoperatorimportitemgetter
>>>a=[(5,3),(1,3),(1,2),(2,-1),(4,9)]
WOW! eBook
www.wowebook.org
>>>sorted(a)
[(1,2),(1,3),(2,-1),(4,9),(5,3)]
>>>sorted(a,key=itemgetter(0))
[(1,3),(1,2),(2,-1),(4,9),(5,3)]
>>>sorted(a,key=itemgetter(0,1))
[(1,2),(1,3),(2,-1),(4,9),(5,3)]
>>>sorted(a,key=itemgetter(1))
[(2,-1),(1,2),(5,3),(1,3),(4,9)]
>>>sorted(a,key=itemgetter(1),reverse=True)
[(4,9),(5,3),(1,3),(1,2),(2,-1)]
Theprecedingcodedeservesalittleexplanation.Firstofall,aisalistoftuples.This
meanseachelementinaisatuple(a2-tuple,tobepicky).Whenwecall
sorted(some_list),wegetasortedversionofsome_list.Inthiscase,thesortingona2tupleworksbysortingthemonthefirstiteminthetuple,andonthesecondwhenthefirst
oneisthesame.Youcanseethisbehaviorintheresultofsorted(a),whichyields[(1,
2),(1,3),...].Pythonalsogivesustheabilitytocontrolonwhichelement(s)ofthe
tuplethesortingmustberunagainst.Noticethatwhenweinstructthesortedfunctionto
workonthefirstelementofeachtuple(bykey=itemgetter(0)),theresultisdifferent:
[(1,3),(1,2),...].Thesortingisdoneonlyonthefirstelementofeachtuple
(whichistheoneatposition0).Ifwewanttoreplicatethedefaultbehaviorofasimple
sorted(a)call,weneedtousekey=itemgetter(0,1),whichtellsPythontosortfirston
theelementsatposition0withinthetuples,andthenonthoseatposition1.Comparethe
resultsandyou’llseetheymatch.
Forcompleteness,Iincludedanexampleofsortingonlyontheelementsatposition1,and
thesamebutinreverseorder.IfyouhaveeverseensortinginJava,Iexpectyoutobeon
yourkneescryingwithjoyatthisverymoment.
ThePythonsortingalgorithmisverypowerful,anditwaswrittenbyTimPeters(we’ve
alreadyseenthisname,canyourecallwhen?).ItisaptlynamedTimsort,anditisablend
betweenmergeandinsertionsortandhasbettertimeperformancesthanmostother
algorithmsusedformainstreamprogramminglanguages.Timsortisastablesorting
algorithm,whichmeansthatwhenmultiplerecordshavethesamekey,theiroriginalorder
ispreserved.We’veseenthisintheresultofsorted(a,key=itemgetter(0))whichhas
yielded[(1,3),(1,2),...]inwhichtheorderofthosetwotupleshasbeen
preservedbecausetheyhavethesamevalueatposition0.
WOW! eBook
www.wowebook.org
Bytearrays
Toconcludeouroverviewofmutablesequencetypes,let’sspendacoupleofminuteson
thebytearraytype.Basically,theyrepresentthemutableversionofbytesobjects.They
exposemostoftheusualmethodsofmutablesequencesaswellasmostofthemethodsof
thebytestype.Itemsareintegersintherange[0,256).
Note
Whenitcomestointervals,I’mgoingtousethestandardnotationforopen/closedranges.
Asquarebracketononeendmeansthatthevalueisincluded,whilearoundbracemeans
it’sexcluded.Thegranularityisusuallyinferredbythetypeoftheedgeelementsso,for
example,theinterval[3,7]meansallintegersbetween3and7,inclusive.Ontheother
hand,(3,7)meansallintegersbetween3and7exclusive(hence4,5,and6).Itemsina
bytearraytypeareintegersbetween0and256,0isincluded,256isnot.Onereason
intervalsareoftenexpressedlikethisistoeasecoding.Ifwebreakarange[a,b)intoN
consecutiveranges,wecaneasilyrepresenttheoriginaloneasaconcatenationlikethis:
Themiddlepoints(ki)beingexcludedononeend,andincludedontheotherend,allow
foreasyconcatenationandsplittingwhenintervalsarehandledinthecode.
Let’sseeaquickexamplewiththetypebytearray:
>>>bytearray()#emptybytearrayobject
bytearray(b'')
>>>bytearray(10)#zero-filledinstancewithgivenlength
bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
>>>bytearray(range(5))#bytearrayfromiterableofintegers
bytearray(b'\x00\x01\x02\x03\x04')
>>>name=bytearray(b'Lina')#A-bytearrayfrombytes
>>>name.replace(b'L',b'l')
bytearray(b'lina')
>>>name.endswith(b'na')
True
>>>name.upper()
bytearray(b'LINA')
>>>name.count(b'L')
1
Asyoucanseeintheprecedingcode,thereareafewwaystocreateabytearrayobject.
Theycanbeusefulinmanysituations,forexample,whenreceivingdatathroughasocket,
theyeliminatetheneedtoconcatenatedatawhilepolling,hencetheyproveveryhandy.
Ontheline#A,Icreatedthenamebytearrayfromthestringb'Lina'toshowyouhowthe
bytearrayobjectexposesmethodsfrombothsequencesandstrings,whichisextremely
handy.Ifyouthinkaboutit,theycanbeconsideredasmutablestrings.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Settypes
Pythonalsoprovidestwosettypes,setandfrozenset.Thesettypeismutable,while
frozensetisimmutable.Theyareunorderedcollectionsofimmutableobjects.
Hashabilityisacharacteristicthatallowsanobjecttobeusedasasetmemberaswellas
akeyforadictionary,aswe’llseeverysoon.
Note
Anobjectishashableifithasahashvaluewhichneverchangesduringitslifetime.
Objectsthatcompareequallymusthavethesamehashvalue.Setsareverycommonly
usedtotestformembership,solet’sintroducetheinoperatorinthefollowingexample:
>>>small_primes=set()#emptyset
>>>small_primes.add(2)#addingoneelementatatime
>>>small_primes.add(3)
>>>small_primes.add(5)
>>>small_primes
{2,3,5}
>>>small_primes.add(1)#LookwhatI'vedone,1isnotaprime!
>>>small_primes
{1,2,3,5}
>>>small_primes.remove(1)#solet'sremoveit
>>>3insmall_primes#membershiptest
True
>>>4insmall_primes
False
>>>4notinsmall_primes#negatedmembershiptest
True
>>>small_primes.add(3)#tryingtoadd3again
>>>small_primes
{2,3,5}#nochange,duplicationisnotallowed
>>>bigger_primes=set([5,7,11,13])#fastercreation
>>>small_primes|bigger_primes#unionoperator`|`
{2,3,5,7,11,13}
>>>small_primes&bigger_primes#intersectionoperator`&`
{5}
>>>small_primes-bigger_primes#differenceoperator`-`
{2,3}
Intheprecedingcode,youcanseetwodifferentwaystocreateaset.Onecreatesan
emptysetandthenaddselementsoneatatime.Theothercreatesthesetusingalistof
numbersasargumenttotheconstructor,whichdoesalltheworkforus.Ofcourse,you
cancreateasetfromalistortuple(oranyiterable)andthenyoucanaddandremove
membersfromthesetasyouplease.
Anotherwayofcreatingasetisbysimplyusingthecurlybracesnotation,likethis:
>>>small_primes={2,3,5,5,3}
>>>small_primes
{2,3,5}
WOW! eBook
www.wowebook.org
NoticeIaddedsomeduplicationtoemphasizethattheresultsetwon’thaveany.
Note
We’llseeiterableobjectsanditerationinthenextchapter.Fornow,justknowthatiterable
objectsareobjectsyoucaniterateoninadirection.
Let’sseeanexampleabouttheimmutablecounterpartofthesettype:frozenset.
>>>small_primes=frozenset([2,3,5,7])
>>>bigger_primes=frozenset([5,7,11])
>>>small_primes.add(11)#wecannotaddtoafrozenset
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
AttributeError:'frozenset'objecthasnoattribute'add'
>>>small_primes.remove(2)#neitherwecanremove
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
AttributeError:'frozenset'objecthasnoattribute'remove'
>>>small_primes&bigger_primes#intersect,union,etc.allowed
frozenset({5,7})
Asyoucansee,frozensetobjectsarequitelimitedinrespectoftheirmutable
counterpart.Theystillproveveryeffectiveformembershiptest,union,intersectionand
differenceoperations,andforperformancereasons.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Mappingtypes–dictionaries
Ofallthebuilt-inPythondatatypes,thedictionaryisprobablythemostinterestingone.
It’stheonlystandardmappingtype,anditisthebackboneofeveryPythonobject.
Adictionarymapskeystovalues.Keysneedtobehashableobjects,whilevaluescanbe
ofanyarbitrarytype.Dictionariesaremutableobjects.
Therearequiteafewdifferentwaystocreateadictionary,soletmegiveyouasimple
exampleofhowtocreateadictionaryequalto{'A':1,'Z':-1}infivedifferentways:
>>>a=dict(A=1,Z=-1)
>>>b={'A':1,'Z':-1}
>>>c=dict(zip(['A','Z'],[1,-1]))
>>>d=dict([('A',1),('Z',-1)])
>>>e=dict({'Z':-1,'A':1})
>>>a==b==c==d==e#aretheyallthesame?
True#indeed!
Haveyounoticedthosedoubleequals?Assignmentisdonewithoneequal,whiletocheck
whetheranobjectisthesameasanotherone(or5inonego,inthiscase),weusedouble
equals.Thereisalsoanotherwaytocompareobjects,whichinvolvestheisoperator,and
checkswhetherthetwoobjectsarethesame(iftheyhavethesameID,notjustthevalue),
butunlessyouhaveagoodreasontouseit,youshouldusethedoubleequalinstead.In
theprecedingcode,Ialsousedonenicefunction:zip.Itisnamedafterthereal-lifezip,
whichgluestogethertwothingstakingoneelementfromeachatatime.Letmeshowyou
anexample:
>>>list(zip(['h','e','l','l','o'],[1,2,3,4,5]))
[('h',1),('e',2),('l',3),('l',4),('o',5)]
>>>list(zip('hello',range(1,6)))#equivalent,morePythonic
[('h',1),('e',2),('l',3),('l',4),('o',5)]
Intheprecedingexample,Ihavecreatedthesamelistintwodifferentways,onemore
explicit,andtheotheralittlebitmorePythonic.ForgetforamomentthatIhadtowrap
thelistconstructoraroundthezipcall(thereasonisbecausezipreturnsaniterator,not
alist),andconcentrateontheresult.Seehowziphascoupledthefirstelementsofits
twoargumentstogether,thenthesecondones,thenthethirdones,andsoonandsoforth?
Takealookatyourpants(oratyourpurseifyou’realady)andyou’llseethesame
behaviorinyouractualzip.Butlet’sgobacktodictionariesandseehowmanywonderful
methodstheyexposeforallowingustomanipulatethemaswewant.Let’sstartwiththe
basicoperations:
>>>d={}
>>>d['a']=1#let'ssetacoupleof(key,value)pairs
>>>d['b']=2
>>>len(d)#howmanypairs?
2
>>>d['a']#whatisthevalueof'a'?
1
>>>d#howdoes`d`looknow?
WOW! eBook
www.wowebook.org
{'a':1,'b':2}
>>>deld['a']#let'sremove`a`
>>>d
{'b':2}
>>>d['c']=3#let'sadd'c':3
>>>'c'ind#membershipischeckedagainstthekeys
True
>>>3ind#notthevalues
False
>>>'e'ind
False
>>>d.clear()#let'scleaneverythingfromthisdictionary
>>>d
{}
Noticehowaccessingkeysofadictionary,regardlessofthetypeofoperationwe’re
performing,isdonethroughsquarebrackets.Doyourememberstrings,list,andtuples?
Wewereaccessingelementsatsomepositionthroughsquarebracketsaswell.Yetanother
exampleofPython’sconsistency.
Let’sseenowthreespecialobjectscalleddictionaryviews:keys,values,anditems.
Theseobjectsprovideadynamicviewofthedictionaryentriesandtheychangewhenthe
dictionarychanges.keys()returnsallthekeysinthedictionary,values()returnsallthe
valuesinthedictionary,anditems()returnsallthe(key,value)pairsinthedictionary.
Note
It’sveryimportanttoknowthat,evenifadictionaryisnotintrinsicallyordered,according
tothePythondocumentation:“Keysandvaluesareiteratedoverinanarbitraryorder
whichisnon-random,variesacrossPythonimplementations,anddependsonthe
dictionary’shistoryofinsertionsanddeletions.Ifkeys,valuesanditemsviewsareiterated
overwithnointerveningmodificationstothedictionary,theorderofitemswilldirectly
correspond.”
Enoughwiththischatter,let’sputallthisdownintocode:
>>>d=dict(zip('hello',range(5)))
>>>d
{'e':1,'h':0,'o':4,'l':3}
>>>d.keys()
dict_keys(['e','h','o','l'])
>>>d.values()
dict_values([1,0,4,3])
>>>d.items()
dict_items([('e',1),('h',0),('o',4),('l',3)])
>>>3ind.values()
True
>>>('o',4)ind.items()
True
Afewthingstonoticeintheprecedingcode.First,noticehowwe’recreatingadictionary
byiteratingoverthezippedversionofthestring'hello'andthelist[0,1,2,3,4].
Thestring'hello'hastwo'l'charactersinside,andtheyarepairedupwiththevalues2
WOW! eBook
www.wowebook.org
and3bythezipfunction.Noticehowinthedictionary,thesecondoccurrenceofthe'l'
key(theonewithvalue3),overwritesthefirstone(theonewithvalue2).Anotherthingto
noticeisthatwhenaskingforanyview,theoriginalorderislost,butisconsistentwithin
theviews,asexpected.Noticealsothatyoumayhavedifferentresultswhenyoutrythis
codeonyourmachine.Pythondoesn’tguaranteethat,itonlyguaranteestheconsistencyof
theorderinwhichtheviewsarepresented.
We’llseehowtheseviewsarefundamentaltoolswhenwetalkaboutiteratingover
collections.Let’stakealooknowatsomeothermethodsexposedbyPython’s
dictionaries,there’splentyofthemandtheyareveryuseful:
>>>d
{'e':1,'h':0,'o':4,'l':3}
>>>d.popitem()#removesarandomitem
('e',1)
>>>d
{'h':0,'o':4,'l':3}
>>>d.pop('l')#removeitemwithkey`l`
3
>>>d.pop('not-a-key')#removeakeynotindictionary:KeyError
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
KeyError:'not-a-key'
>>>d.pop('not-a-key','default-value')#withadefaultvalue?
'default-value'#wegetthedefaultvalue
>>>d.update({'another':'value'})#wecanupdatedictthisway
>>>d.update(a=13)#orthisway(likeafunctioncall)
>>>d
{'a':13,'another':'value','h':0,'o':4}
>>>d.get('a')#sameasd['a']butifkeyismissingnoKeyError
13
>>>d.get('a',177)#defaultvalueusedifkeyismissing
13
>>>d.get('b',177)#likeinthiscase
177
>>>d.get('b')#keyisnotthere,soNoneisreturned
Allthesemethodsarequitesimpletounderstand,butit’sworthtalkingaboutthatNone,
foramoment.EveryfunctioninPythonreturnsNone,unlessthereturnstatementis
explicitlyused,butwe’llseethiswhenweexplorefunctions.Noneisfrequentlyusedto
representtheabsenceofavalue,aswhendefaultargumentsarenotpassedtoafunction.
SomeinexperiencedcoderssometimeswritecodethatreturnseitherFalseorNone.Both
FalseandNoneevaluatetoFalsesoitmayseemthereisnotmuchdifferencebetween
them.Butactually,Iwouldarguethereisquiteanimportantdifference:Falsemeansthat
wehaveinformation,andtheinformationwehaveisFalse.Nonemeansnoinformation.
Andnoinformationisverydifferentfromaninformation,whichisFalse.Inlayman’s
terms,ifyouaskyourmechanic“ismycarready?”thereisabigdifferencebetweenthe
answer“No,it’snot”(False)and“Ihavenoidea”(None).
OnelastmethodIreallylikeofdictionariesissetdefault.Itbehaveslikeget,butalso
setsthekeywiththegivenvalueifitisnotthere.Let’sseeandexample:
WOW! eBook
www.wowebook.org
>>>d={}
>>>d.setdefault('a',1)#'a'ismissing,wegetdefaultvalue
1
>>>d
{'a':1}#also,thekey/valuepair('a',1)hasnowbeenadded
>>>d.setdefault('a',5)#let'strytooverridethevalue
1
>>>d
{'a':1}#didn'twork,asexpected
So,we’renowattheendofthistour.Testyourknowledgeaboutdictionariestryingto
foreseehowdlookslikeafterthisline.
>>>d={}
>>>d.setdefault('a',{}).setdefault('b',[]).append(1)
It’snotthatcomplicated,butdon’tworryifyoudon’tgetitimmediately.Ijustwantedto
spuryoutoexperimentwithdictionaries.
Thisconcludesourtourofbuilt-indatatypes.BeforeImakesomeconsiderationsabout
whatwe’veseeninthischapter,Iwanttobrieflytakeapeekatthecollectionsmodule.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Thecollectionsmodule
WhenPythongeneralpurposebuilt-incontainers(tuple,list,set,anddict)aren’t
enough,wecanfindspecializedcontainerdatatypesinthecollectionsmodule.They
are:
Datatype
Description
namedtuple()
Afactoryfunctionforcreatingtuplesubclasseswithnamedfields
deque
Alist-likecontainerwithfastappendsandpopsoneitherend
ChainMap
Adict-likeclassforcreatingasingleviewofmultiplemappings
Counter
Adictsubclassforcountinghashableobjects
OrderedDict
Adictsubclassthatrememberstheorderentrieswereadded
defaultdict
Adictsubclassthatcallsafactoryfunctiontosupplymissingvalues
UserDict
Awrapperarounddictionaryobjectsforeasierdictsubclassing
UserList
Awrapperaroundlistobjectsforeasierlistsubclassing
UserString
Awrapperaroundstringobjectsforeasierstringsubclassing
Wedon’thavetheroomtocoverallofthem,butyoucanfindplentyofexamplesinthe
officialdocumentation,sohereI’lljustgiveasmallexampletoshowyounamedtuple,
defaultdict,andChainMap.
WOW! eBook
www.wowebook.org
Namedtuples
Anamedtupleisatuple-likeobjectthathasfieldsaccessiblebyattributelookupaswell
asbeingindexableanditerable(it’sactuallyasubclassoftuple).Thisissortofa
compromisebetweenafull-fledgedobjectandatuple,anditcanbeusefulinthosecases
whereyoudon’tneedthefullpowerofacustomobject,butyouwantyourcodetobe
morereadablebyavoidingweirdindexing.Anotherusecaseiswhenthereisachancethat
itemsinthetupleneedtochangetheirpositionafterrefactoring,forcingthecoderto
refactoralsoallthelogicinvolved,whichcanbeverytricky.Asusual,anexampleis
betterthanathousandwords(orwasitapicture?).Saywearehandlingdataabouttheleft
andrighteyeofapatient.Wesaveonevalueforthelefteye(position0)andoneforthe
righteye(position1)inaregulartuple.Here’showthatmightbe:
>>>vision=(9.5,8.8)
>>>vision
(9.5,8.8)
>>>vision[0]#lefteye(implicitpositionalreference)
9.5
>>>vision[1]#righteye(implicitpositionalreference)
8.8
Nowlet’spretendwehandlevisionobjectallthetime,andatsomepointthedesigner
decidestoenhancethembyaddinginformationforthecombinedvision,sothatavision
objectstoresdatainthisformat:(lefteye,combined,righteye).
Doyouseethetroublewe’reinnow?Wemayhavealotofcodethatdependson
vision[0]beingthelefteyeinformation(whichstillis)andvision[1]beingtheright
eyeinformation(whichisnolongerthecase).Wehavetorefactorourcodewhereverwe
handletheseobjects,changingvision[1]tovision[2],anditcanbepainful.Wecould
haveprobablyapproachedthisabitbetterfromthebeginning,byusinganamedtuple.Let
meshowyouwhatImean:
>>>fromcollectionsimportnamedtuple
>>>Vision=namedtuple('Vision',['left','right'])
>>>vision=Vision(9.5,8.8)
>>>vision[0]
9.5
>>>vision.left#sameasvision[0],butexplicit
9.5
>>>vision.right#sameasvision[1],butexplicit
8.8
Ifwithinourcodewerefertoleftandrighteyeusingvision.leftandvision.right,all
weneedtodotofixthenewdesignissueistochangeourfactoryandthewaywecreate
instances.Therestofthecodewon’tneedtochange.
>>>Vision=namedtuple('Vision',['left','combined','right'])
>>>vision=Vision(9.5,9.2,8.8)
>>>vision.left#stillperfect
9.5
>>>vision.right#stillperfect(thoughnowisvision[2])
WOW! eBook
www.wowebook.org
8.8
>>>vision.combined#thenewvision[1]
9.2
Youcanseehowconvenientitistorefertothosevaluesbynameratherthanbyposition.
Afterall,awisemanoncewrote“Explicitisbetterthanimplicit”(canyourecallwhere?
Thinkzenifyoudon’t…).Thisexamplemaybealittleextreme,ofcourseit’snotlikely
thatourcodedesignerwillgoforachangelikethis,butyou’dbeamazedtoseehow
frequentlyissuessimilartothisonehappeninaprofessionalenvironment,andhow
painfulitistorefactorthem.
WOW! eBook
www.wowebook.org
Defaultdict
Thedefaultdictdatatypeisoneofmyfavorites.Itallowsyoutoavoidcheckingifakey
isinadictionarybysimplyinsertingitforyouonyourfirstaccessattempt,withadefault
valuewhosetypeyoupassoncreation.Insomecases,thistoolcanbeveryhandyand
shortenyourcodealittle.Let’sseeaquickexample:sayweareupdatingthevalueofage,
byaddingoneyear.Ifageisnotthere,weassumeitwas0andweupdateitto1.
>>>d={}
>>>d['age']=d.get('age',0)+1#agenotthere,weget0+1
>>>d
{'age':1}
>>>d={'age':39}
>>>d['age']=d.get('age',0)+1#disthere,weget40
>>>d
{'age':40}
Nowlet’sseehowitwouldworkwithadefaultdictdatatype.Thesecondlineis
actuallytheshortversionofa4-lineslongifclausethatwewouldhavetowriteif
dictionariesdidn’thavethegetmethod.We’llseeallaboutifclausesinChapter3,
IteratingandMakingDecisions.
>>>fromcollectionsimportdefaultdict
>>>dd=defaultdict(int)#intisthedefaulttype(0thevalue)
>>>dd['age']+=1#shortfordd['age']=dd['age']+1
>>>dd
defaultdict(<class'int'>,{'age':1})#1,asexpected
>>>dd['age']=39
>>>dd['age']+=1
>>>dd
defaultdict(<class'int'>,{'age':40})#40,asexpected
Noticehowwejustneedtoinstructthedefaultdictfactorythatwewantanintnumber
tobeusedincasethekeyismissing(we’llget0,whichisthedefaultfortheinttype).
Also,noticethateventhoughinthisexamplethereisnogainonthenumberoflines,there
isdefinitelyagaininreadability,whichisveryimportant.Youcanalsouseadifferent
techniquetoinstantiateadefaultdictdatatype,whichinvolvescreatingafactoryobject.
Fordiggingdeeper,pleaserefertotheofficialdocumentation.
WOW! eBook
www.wowebook.org
ChainMap
TheChainMapisanextremelynicedatatypewhichwasintroducedinPython3.3.It
behaveslikeanormaldictionarybutaccordingtothePythondocumentation:isprovided
forquicklylinkinganumberofmappingssotheycanbetreatedasasingleunit.Thisis
usuallymuchfasterthancreatingonedictionaryandrunningmultipleupdatecallsonit.
ChainMapcanbeusedtosimulatenestedscopesandisusefulintemplating.The
underlyingmappingsarestoredinalist.Thatlistispublicandcanbeaccessedorupdated
usingthemapsattribute.Lookupssearchtheunderlyingmappingssuccessivelyuntilakey
isfound.Incontrast,writes,updates,anddeletionsonlyoperateonthefirstmapping.
Averycommonusecaseisprovidingdefaults,solet’sseeanexample:
>>>fromcollectionsimportChainMap
>>>default_connection={'host':'localhost','port':4567}
>>>connection={'port':5678}
>>>conn=ChainMap(connection,default_connection)#mapcreation
>>>conn['port']#portisfoundinthefirstdictionary
5678
>>>conn['host']#hostisfetchedfromtheseconddictionary
'localhost'
>>>conn.maps#wecanseethemappingobjects
[{'port':5678},{'host':'localhost','port':4567}]
>>>conn['host']='packtpub.com'#let'saddhost
>>>conn.maps
[{'host':'packtpub.com','port':5678},
{'host':'localhost','port':4567}]
>>>delconn['port']#let'sremovetheportinformation
>>>conn.maps
[{'host':'packtpub.com'},
{'host':'localhost','port':4567}]
>>>conn['port']#nowportisfetchedfromtheseconddictionary
4567
>>>dict(conn)#easytomergeandconverttoregulardictionary
{'host':'packtpub.com','port':4567}
IjustlovehowPythonmakesyourlifeeasy.YouworkonaChainMapobject,configure
thefirstmappingasyouwant,andwhenyouneedacompletedictionarywithallthe
defaultsaswellasthecustomizeditems,youjustfeedtheChainMapobjecttoadict
constructor.Ifyouhavenevercodedinotherlanguages,suchasJavaorC++,you
probablywon’tbeabletofullyappreciatehowpreciousthisis,howPythonmakesyour
lifesomucheasier.Ido,IfeelclaustrophobiceverytimeIhavetocodeinsomeother
language.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Finalconsiderations
That’sit.Nowyouhaveseenaverygoodportionofthedatastructuresthatyouwillusein
Python.IencourageyoutotakeadiveintothePythondocumentationandexperiment
furtherwitheachandeverydatatypewe’veseeninthischapter.It’sworthit,believeme.
Everythingyou’llwritewillbeabouthandlingdata,somakesureyourknowledgeaboutit
isrocksolid.
Beforeweleapintothenextchapter,I’dliketomakesomefinalconsiderationsabout
differentaspectsthattomymindareimportantandnottobeneglected.
WOW! eBook
www.wowebook.org
Smallvaluescaching
Whenwediscussedobjectsatthebeginningofthischapter,wesawthatwhenweassigned
anametoanobject,Pythoncreatestheobject,setsitsvalue,andthenpointsthenameto
it.Wecanassigndifferentnamestothesamevalueandweexpectdifferentobjectstobe
created,likethis:
>>>a=1000000
>>>b=1000000
>>>id(a)==id(b)
False
Intheprecedingexample,aandbareassignedtotwointobjects,whichhavethesame
valuebuttheyarenotthesameobject,asyoucansee,theiridisnotthesame.Solet’sdo
itagain:
>>>a=5
>>>b=5
>>>id(a)==id(b)
True
Ohoh!IsPythonbroken?Whyarethetwoobjectsthesamenow?Wedidn’tdoa=b=
5,wesetthemupseparately.Well,theanswerisperformances.Pythoncachesshort
stringsandsmallnumbers,toavoidhavingmanycopiesofthemcloggingupthesystem
memory.Everythingishandledproperlyunderthehoodsoyoudon’tneedtoworryabit,
butmakesurethatyourememberthisbehaviorshouldyourcodeeverneedtofiddlewith
IDs.
WOW! eBook
www.wowebook.org
Howtochoosedatastructures
Aswe’veseen,Pythonprovidesyouwithseveralbuilt-indatatypesandsometimes,if
you’renotthatexperienced,choosingtheonethatservesyoubestcanbetricky,especially
whenitcomestocollections.Forexample,sayyouhavemanydictionariestostore,each
ofwhichrepresentsacustomer.Withineachcustomerdictionarythere’san'id':'code'
uniqueidentificationcode.Inwhatkindofcollectionwouldyouplacethem?Well,unless
Iknowmoreaboutthesecustomers,it’sveryhardtoanswer.WhatkindofaccesswillI
need?WhatsortofoperationswillIhavetoperformoneachofthem,andhowmany
times?Willthecollectionchangeovertime?WillIneedtomodifythecustomer
dictionariesinanyway?WhatisgoingtobethemostfrequentoperationIwillhaveto
performonthecollection?
Ifyoucananswertheprecedingquestions,thenyouwillknowwhattochoose.Ifthe
collectionnevershrinksorgrows(inotherwords,itwon’tneedtoadd/deleteany
customerobjectaftercreation)orshuffles,thentuplesareapossiblechoice.Otherwise
listsareagoodcandidate.Everycustomerdictionaryhasauniqueidentifierthough,so
evenadictionarycouldwork.Letmedrafttheseoptionsforyou:
#examplecustomerobjects
customer1={'id':'abc123','full_name':'MasterYoda'}
customer2={'id':'def456','full_name':'Obi-WanKenobi'}
customer3={'id':'ghi789','full_name':'AnakinSkywalker'}
#collecttheminatuple
customers=(customer1,customer2,customer3)
#orcollecttheminalist
customers=[customer1,customer2,customer3]
#ormaybewithinadictionary,theyhaveauniqueidafterall
customers={
'abc123':customer1,
'def456':customer2,
'ghi789':customer3,
}
Somecustomerswehavethere,right?Iprobablywouldn’tgowiththetupleoption,unless
Iwantedtohighlightthatthecollectionisnotgoingtochange.I’dsayusuallyalistis
better,itallowsformoreflexibility.
Anotherfactortokeepinmindisthattuplesandlistsareorderedcollections,whileifyou
useadictionaryorasetyoulosetheordering,soyouneedtoknowiforderingis
importantinyourapplication.
Whataboutperformances?Forexampleinalist,operationssuchasinsertionand
membershipcantakeO(n),whiletheyareO(1)foradictionary.It’snotalwayspossibleto
usedictionariesthough,ifwedon’thavetheguaranteethatwecanuniquelyidentifyeach
itemofthecollectionbymeansofoneofitsproperties,andthatthepropertyinquestionis
hashable(soitcanbeakeyindict).
Note
Ifyou’rewonderingwhatO(n)andO(1)mean,pleaseGoogle“bigOnotation”andgeta
WOW! eBook
www.wowebook.org
gistofitfromanywhere.Inthiscontext,let’sjustsaythatifperforminganoperationOp
to
onadatastructuretakesO(f(n)),itwouldmeanthatOptakesatmostatime
complete,wherecissomepositiveconstant,nisthesizeoftheinput,andfissome
function.So,thinkofO(…)asanupperboundfortherunningtimeofanoperation(itcan
beusedalsotosizeothermeasurablequantities,ofcourse).
Anotherwayofunderstandingifyouhavechosentherightdatastructureisbylookingat
thecodeyouhavetowriteinordertomanipulateit.Ifeverythingcomeseasilyandflows
naturally,thenyouprobablyhavechosencorrectly,butifyoufindyourselfthinkingyour
codeisgettingunnecessarilycomplicated,thenyouprobablyshouldtryanddecide
whetheryouneedtoreconsideryourchoices.It’squitehardtogiveadvicewithouta
practicalcasethough,sowhenyouchooseadatastructureforyourdata,trytokeepease
ofuseandperformanceinmindandgiveprecedencetowhatmattersmostinthecontext
youare.
WOW! eBook
www.wowebook.org
Aboutindexingandslicing
Atthebeginningofthischapter,wesawslicingappliedonstrings.Slicingingeneral
appliestoasequence,sotuples,lists,strings,etc.Withlists,slicingcanalsobeusedfor
assignment.I’vealmostneverseenthisusedinprofessionalcode,butstill,youknowyou
can.Couldyouslicedictionariesorsets?Ihearyouscream“Ofcoursenot!Theyarenot
ordered!“.Excellent,Iseewe’reonthesamepagehere,solet’stalkaboutindexing.
ThereisonecharacteristicaboutPythonindexingIhaven’tmentionedbefore.I’llshow
youbyexample.Howdoyouaddressthelastelementofacollection?Let’ssee:
>>>a=list(range(10))#`a`has10elements.Lastoneis9.
>>>a
[0,1,2,3,4,5,6,7,8,9]
>>>len(a)#itslengthis10elements
10
>>>a[len(a)-1]#positionoflastoneislen(a)-1
9
>>>a[-1]#butwedon'tneedlen(a)!Pythonrocks!
9
>>>a[-2]#equivalenttolen(a)-2
8
>>>a[-3]#equivalenttolen(a)-3
7
Ifthelistahas10elements,becauseofthe0-indexpositioningsystemofPython,thefirst
oneisatposition0andthelastoneisatposition9.Intheprecedingexample,theelements
areconvenientlyplacedinapositionequaltotheirvalue:0isatposition0,1atposition1,
andsoon.
So,inordertofetchthelastelement,weneedtoknowthelengthofthewholelist(or
tuple,orstring,andsoon)andthensubtract1.Hence:len(a)–1.Thisissocommonan
operationthatPythonprovidesyouwithawaytoretrieveelementsusingnegative
indexing.Thisprovesveryusefulwhenyoudosomeseriousdatamanipulation.Here’sa
nicediagramabouthowindexingworksonthestring"HelloThere":
Tryingtoaddressindexesgreaterthan9orsmallerthan-10willraiseanIndexError,as
expected.
WOW! eBook
www.wowebook.org
Aboutthenames
Youmayhavenoticedthat,inordertokeeptheexampleasshortaspossible,Ihavecalled
manyobjectsusingsimpleletters,likea,b,c,d,andsoon.Thisisperfectlyokwhenyou
debugontheconsoleorwhenyoushowthata+b==7,butit’sbadpracticewhenit
comestoprofessionalcoding(oranytypeofcoding,forallthatmatter).Ihopeyouwill
indulgemeifIsometimesdoit,thereasonistopresentthecodeinamorecompactway.
Inarealenvironmentthough,whenyouchoosenamesforyourdata,youshouldchoose
themcarefullyandtheyshouldreflectwhatthedataisabout.So,ifyouhaveacollection
ofCustomerobjects,customersisaperfectlygoodnameforit.Wouldcustomers_list,
customers_tuple,orcustomers_collectionworkaswell?Thinkaboutitforasecond.
Isitgoodtotiethenameofthecollectiontothedatatype?Idon’tthinkso,atleastin
mostcases.SoI’dsayifyouhaveanexcellentreasontodosogoahead,otherwisedon’t.
Thereasonis,oncethatcustomers_tuplestartsbeingusedindifferentplacesofyour
code,andyourealizeyouactuallywanttousealistinsteadofatuple,you’reupforsome
funrefactoring(alsoknownaswastedtime).Namesfordatashouldbenouns,andnames
forfunctionsshouldbeverbs.Namesshouldbeasexpressiveaspossible.Pythonis
actuallyaverygoodexamplewhenitcomestonames.Mostofthetimeyoucanjustguess
whatafunctioniscalledifyouknowwhatitdoes.Crazy,huh?
Chapter2,MeaningfulNamesofCleanCode,RobertC.Martin,PrenticeHallisentirely
dedicatedtonames.It’sanamazingbookthathelpedmeimprovemycodingstylein
manydifferentways,amustreadifyouwanttotakeyourcodingtothenextlevel.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,we’veexploredthebuilt-indatatypesofPython.We’veseenhowmany
theyareandhowmuchcanbeachievedbyjustusingthemindifferentcombinations.
We’veseennumbertypes,sequences,sets,mappings,collections,we’veseenthat
everythingisanobject,we’velearnedthedifferencebetweenmutableandimmutable,and
we’vealsolearnedaboutslicingandindexing(and,proudly,negativeindexingaswell).
We’vepresentedsimpleexamples,butthere’smuchmorethatyoucanlearnaboutthis
subject,sostickyournoseintotheofficialdocumentationandexplore.
Mostofall,Iencourageyoutotryoutalltheexercisesbyyourself,getyourfingersusing
thatcode,buildsomemusclememory,andexperiment,experiment,experiment.Learn
whathappenswhenyoudividebyzero,whenyoucombinedifferentnumbertypesintoa
singleexpression,whenyoumanagestrings.Playwithalldatatypes.Exercisethem,break
them,discoveralltheirmethods,enjoythemandlearnthemwell,damnwell.
Ifyourfoundationisnotrocksolid,howgoodcanyourcodebe?Anddataisthe
foundationforeverything.Datashapeswhatdancesaroundit.
Themoreyouprogresswiththebook,themoreit’slikelythatyouwillfindsome
discrepanciesormaybeasmalltypohereandthereinmycode(oryours).Youwillgetan
errormessage,somethingwillbreak.That’swonderful!Whenyoucode,thingsbreakall
thetime,youdebugandfixallthetime,soconsidererrorsasusefulexercisestolearn
somethingnewaboutthelanguageyou’reusing,andnotasfailuresorproblems.Errors
willkeepcomingupuntilyourverylastlineofcode,that’sforsure,soyoumayaswell
startmakingyourpeacewiththemnow.
Thenextchapterisaboutiteratingandmakingdecisions.We’llseehowtoactuallyput
thosecollectionsinuse,andtakedecisionsbasedonthedatawe’representedwith.We’ll
starttogoalittlefasternowthatyourknowledgeisbuildingup,somakesureyou’re
comfortablewiththecontentsofthischapterbeforeyoumovetothenextone.Oncemore,
havefun,explore,breakthings.It’saverygoodwaytolearn.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter3.IteratingandMaking
Decisions
“Insanity:doingthesamethingoverandoveragainandexpectingdifferentresults.” —AlbertEinstein
Inthepreviouschapter,we’veseenPythonbuilt-indatatypes.Nowthatyou’refamiliar
withdatainitsmanyformsandshapes,it’stimetostartlookingathowaprogramcanuse
it.
AccordingtoWikipedia:
Incomputerscience,controlflow(oralternatively,flowofcontrol)referstothe
specificationoftheorderinwhichtheindividualstatements,instructionsorfunction
callsofanimperativeprogramareexecutedorevaluated.
Inordertocontroltheflowofaprogram,wehavetwomainweapons:conditional
programming(alsoknownasbranching)andlooping.Wecanusetheminmany
differentcombinationsandvariations,butinthischapter,insteadofgoingthroughall
possiblevariousformsofthosetwoconstructsina“documentation”fashion,I’drather
giveyouthebasicsandthenI’llwriteacoupleofsmallscriptswithyou.Inthefirstone,
we’llseehowtocreatearudimentaryprimenumbergenerator,whileinthesecondone,
we’llseehowtoapplydiscountstocustomersbasedoncoupons.Thiswayyoushouldget
abetterfeelingabouthowconditionalprogrammingandloopingcanbeused.
WOW! eBook
www.wowebook.org
Conditionalprogramming
Conditionalprogramming,orbranching,issomethingyoudoeveryday,everymoment.
It’saboutevaluatingconditions:ifthelightisgreen,thenIcancross,ifit’sraining,then
I’mtakingtheumbrella,andifI’mlateforwork,thenI’llcallmymanager.
Themaintoolistheifstatement,whichcomesindifferentformsandcolors,butbasically
whatitdoesisevaluateanexpressionand,basedontheresult,choosewhichpartofthe
codetoexecute.Asusual,let’sseeanexample:
conditional.1.py
late=True
iflate:
print('Ineedtocallmymanager!')
Thisispossiblythesimplestexample:whenfedtotheifstatement,lateactsasa
conditionalexpression,whichisevaluatedinaBooleancontext(exactlylikeifwewere
callingbool(late)).IftheresultoftheevaluationisTrue,thenweenterthebodyofcode
immediatelyaftertheifstatement.Noticethattheprintinstructionisindented:this
meansitbelongstoascopedefinedbytheifclause.Executionofthiscodeyields:
$pythonconditional.1.py
Ineedtocallmymanager!
SincelateisTrue,theprintstatementwasexecuted.Let’sexpandonthisexample:
conditional.2.py
late=False
iflate:
print('Ineedtocallmymanager!')#1
else:
print('noneedtocallmymanager…')#2
ThistimeIsetlate=False,sowhenIexecutethecode,theresultisdifferent:
$pythonconditional.2.py
noneedtocallmymanager…
Dependingontheresultofevaluatingthelateexpression,wecaneitherenterblock#1or
block#2,butnotboth.Block#1isexecutedwhenlateevaluatestoTrue,whileblock#2
isexecutedwhenlateevaluatestoFalse.TryassigningFalse/Truevaluestothelate
name,andseehowtheoutputforthiscodechangesaccordingly.
Theprecedingexamplealsointroducestheelseclause,whichbecomesveryhandywhen
wewanttoprovideanalternativesetofinstructionstobeexecutedwhenanexpression
evaluatestoFalsewithinanifclause.Theelseclauseisoptional,asit’sevidentby
comparingtheprecedingtwoexamples.
WOW! eBook
www.wowebook.org
Aspecializedelse:elif
Sometimesallyouneedistodosomethingifaconditionismet(simpleifclause).Other
timesyouneedtoprovideanalternative,incasetheconditionisFalse(if/elseclause),
buttherearesituationswhereyoumayhavemorethantwopathstochoosefrom,so,since
callingthemanager(ornotcallingthem)iskindofabinarytypeofexample(eitheryou
calloryoudon’t),let’schangethetypeofexampleandkeepexpanding.Thistimewe
decidetaxpercentages.Ifmyincomeislessthen10k,Iwon’tpayanytaxes.Ifitis
between10kand30k,I’llpay20%taxes.Ifitisbetween30kand100k,I’llpay35%
taxes,andover100k,I’ll(gladly)pay45%taxes.Let’sputthisalldownintobeautiful
Pythoncode:
taxes.py
income=15000
ifincome<10000:
tax_coefficient=0.0#1
elifincome<30000:
tax_coefficient=0.2#2
elifincome<100000:
tax_coefficient=0.35#3
else:
tax_coefficient=0.45#4
print('Iwillpay:',income*tax_coefficient,'intaxes')
Executingtheprecedingcodeyields:
$pythontaxes.py
Iwillpay:3000.0intaxes
Let’sgothroughtheexamplelinebyline:westartbysettinguptheincomevalue.Inthe
example,myincomeis15k.Weentertheifclause.Noticethatthistimewealso
introducedtheelifclause,whichisacontractionforelse-if,andit’sdifferentfroma
bareelseclauseinthatitalsohasitsowncondition.So,theifexpressionincome<
10000,evaluatestoFalse,thereforeblock#1isnotexecuted.Thecontrolpassestothe
nextconditionevaluator:elifincome<30000.ThisoneevaluatestoTrue,therefore
block#2isexecuted,andbecauseofthis,Pythonthenresumesexecutionafterthewhole
if/elif/elif/elseclause(whichwecanjustcallifclausefromnowon).Thereisonly
oneinstructionaftertheifclause,theprintcall,whichtellsusIwillpay3kintaxesthis
year(15k*20%).Noticethattheorderismandatory:ifcomesfirst,then(optionally)as
manyelifasyouneed,andthen(optionally)anelseclause.
Interesting,right?Nomatterhowmanylinesofcodeyoumayhavewithineachblock,
whenoneoftheconditionsevaluatestoTrue,theassociatedblockisexecutedandthen
executionresumesafterthewholeclause.IfnoneoftheconditionsevaluatestoTrue(for
example,income=200000),thenthebodyoftheelseclausewouldbeexecuted(block
#4).Thisexampleexpandsourunderstandingofthebehavioroftheelseclause.Itsblock
ofcodeisexecutedwhennoneoftheprecedingif/elif/…/elifexpressionshas
evaluatedtoTrue.
WOW! eBook
www.wowebook.org
Trytomodifythevalueofincomeuntilyoucancomfortablyexecuteallblocksatyour
will(oneperexecution,ofcourse).Andthentrytheboundaries.Thisiscrucial,
wheneveryouhaveconditionsexpressedasequalitiesorinequalities(==,!=,<,>,<=,
>=),thosenumbersrepresentboundaries.Itisessentialtotestboundariesthoroughly.
ShouldIallowyoutodriveat18or17?AmIcheckingyouragewithage<18,orage
<=18?Youcan’timaginehowmanytimesIhadtofixsubtlebugsthatstemmedfrom
usingthewrongoperator,sogoaheadandexperimentwiththeprecedingcode.Change
some<to<=andsetincometobeoneoftheboundaryvalues(10k,30k,100k)aswellas
anyvalueinbetween.Seehowtheresultchanges,getagoodunderstandingofitbefore
proceeding.
Beforewemovetothenexttopic,let’sseeanotherexamplethatshowsushowtonestif
clauses.Sayyourprogramencountersanerror.Ifthealertsystemistheconsole,weprint
theerror.Ifthealertsystemisane-mail,wesenditaccordingtotheseverityoftheerror.
Ifthealertsystemisanythingotherthanconsoleore-mail,wedon’tknowwhattodo,
thereforewedonothing.Let’sputthisintocode:
errorsalert.py
alert_system='console'#othervaluecanbe'email'
error_severity='critical'#othervalues:'medium'or'low'
error_message='OMG!Somethingterriblehappened!'
ifalert_system=='console':
print(error_message)#1
elifalert_system=='email':
iferror_severity=='critical':
send_email('[email protected]',error_message)#2
eliferror_severity=='medium':
send_email('[email protected]',error_message)#3
else:
send_email('[email protected]',error_message)#4
Theprecedingexampleisquiteinteresting,initssilliness.Itshowsustwonestedif
clauses(outerandinner).Italsoshowsustheouterifclausedoesn’thaveanyelse,
whiletheinneronedoes.Noticehowindentationiswhatallowsustonestoneclause
withinanotherone.
Ifalert_system=='console',body#1isexecuted,andnothingelsehappens.Onthe
otherhand,ifalert_system=='email',thenweenterintoanotherifclause,whichwe
calledinner.Intheinnerifclause,accordingtoerror_severity,wesendane-mailto
eitheranadmin,first-levelsupport,orsecond-levelsupport(blocks#2,#3,and#4).The
send_emailfunctionisnotdefinedinthisexample,thereforetryingtorunitwouldgive
youanerror.Inthesourcecodeofthebook,whichyoucandownloadfromthewebsite,I
includedatricktoredirectthatcalltoaregularprintfunction,justsoyoucanexperiment
ontheconsolewithoutactuallysendingane-mail.Trychangingthevaluesandseehowit
allworks.
WOW! eBook
www.wowebook.org
Theternaryoperator
OnelastthingIwouldliketoshowyoubeforemovingontothenextsubject,isthe
ternaryoperatoror,inlayman’sterms,theshortversionofanif/elseclause.Whenthe
valueofanameistobeassignedaccordingtosomecondition,sometimesit’seasierand
morereadabletousetheternaryoperatorinsteadofaproperifclause.Inthefollowing
example,thetwocodeblocksdoexactlythesamething:
ternary.py
order_total=247#GBP
#classicif/elseform
iforder_total>100:
discount=25#GBP
else:
discount=0#GBP
print(order_total,discount)
#ternaryoperator
discount=25iforder_total>100else0
print(order_total,discount)
Forsimplecaseslikethis,Ifinditverynicetobeabletoexpressthatlogicinoneline
insteadoffour.Remember,asacoder,youspendmuchmoretimereadingcodethen
writingit,soPythonconcisenessisinvaluable.
Areyouclearonhowtheternaryoperatorworks?Basicallyisname=somethingif
conditionelsesomething-else.Sonameisassignedsomethingifconditionevaluates
toTrue,andsomething-elseifconditionevaluatestoFalse.
Nowthatyouknoweverythingaboutcontrollingthepathofthecode,let’smoveontothe
nextsubject:looping.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Looping
Ifyouhaveanyexperiencewithloopinginotherprogramminglanguages,youwillfind
Python’swayofloopingabitdifferent.Firstofall,whatislooping?Loopingmeansbeing
abletorepeattheexecutionofacodeblockmorethanonce,accordingtotheloop
parameterswe’regiven.Therearedifferentloopingconstructs,whichservedifferent
purposes,andPythonhasdistilledallofthemdowntojusttwo,whichyoucanuseto
achieveeverythingyouneed.Thesearetheforandwhilestatements.
Whileit’sdefinitelypossibletodoeverythingyouneedusingeitherofthem,theyserve
differentpurposesandthereforethey’reusuallyusedindifferentcontexts.We’llexplore
thisdifferencethoroughlythroughthischapter.
WOW! eBook
www.wowebook.org
Theforloop
Theforloopisusedwhenloopingoverasequence,likealist,tuple,oracollectionof
objects.Let’sstartwithasimpleexamplethatismorelikeC++style,andthenlet’s
graduallyseehowtoachievethesameresultsinPython(you’lllovePython’ssyntax).
simple.for.py
fornumberin[0,1,2,3,4]:
print(number)
Thissimplesnippetofcode,whenexecuted,printsallnumbersfrom0to4.Theforloop
isfedthelist[0,1,2,3,4]andateachiteration,numberisgivenavaluefromthe
sequence(whichisiteratedsequentially,inorder),thenthebodyoftheloopisexecuted
(theprintline).numberchangesateveryiteration,accordingtowhichvalueiscoming
nextfromthesequence.Whenthesequenceisexhausted,theforloopterminates,andthe
executionofthecoderesumesnormallywiththecodeaftertheloop.
Iteratingoverarange
Sometimesweneedtoiterateoverarangeofnumbers,anditwouldbequiteunpleasantto
havetodosobyhardcodingthelistsomewhere.Insuchcases,therangefunctioncomes
totherescue.Let’sseetheequivalentoftheprevioussnippetofcode:
simple.for.py
fornumberinrange(5):
print(number)
TherangefunctionisusedextensivelyinPythonprogramswhenitcomestocreating
sequences:youcancallitbypassingonevalue,whichactsasstop(countingfrom0),or
youcanpasstwovalues(startandstop),oreventhree(start,stop,andstep).Check
outthefollowingexample:
>>>list(range(10))#onevalue:from0tovalue(excluded)
[0,1,2,3,4,5,6,7,8,9]
>>>list(range(3,8))#twovalues:fromstarttostop(excluded)
[3,4,5,6,7]
>>>list(range(-10,10,4))#threevalues:stepisadded
[-10,-6,-2,2,6]
Forthemoment,ignorethatweneedtowraprange(...)withinalist.Therangeobject
isalittlebitspecial,butinthiscasewe’rejustinterestedinunderstandingwhatarethe
valuesitwillreturntous.Youseethatthedealisthesamewithslicing:startisincluded,
stopexcluded,andoptionallyyoucanaddastepparameter,whichbydefaultis1.
Trymodifyingtheparametersoftherange()callinoursimple.for.pycodeandsee
whatitprints,getcomfortablewithit.
Iteratingoverasequence
Nowwehaveallthetoolstoiterateoverasequence,solet’sbuildonthatexample:
simple.for.2.py
WOW! eBook
www.wowebook.org
surnames=['Rivest','Shamir','Adleman']
forpositioninrange(len(surnames)):
print(position,surnames[position])
Theprecedingcodeaddsalittlebitofcomplexitytothegame.Executionwillshowthis
result:
$pythonsimple.for.2.py
0Rivest
1Shamir
2Adleman
Let’susetheinside-outtechniquetobreakitdown,ok?Westartfromtheinnermostpart
ofwhatwe’retryingtounderstand,andweexpandoutwards.So,len(surnames)isthe
lengthofthesurnameslist:3.Therefore,range(len(surnames))isactuallytransformed
intorange(3).Thisgivesustherange[0,3),whichisbasicallyasequence(0,1,2).
Thismeansthattheforloopwillrunthreeiterations.Inthefirstone,positionwilltake
value0,whileinthesecondone,itwilltakevalue1,andfinallyvalue2inthethirdand
lastiteration.Whatis(0,1,2),ifnotthepossibleindexingpositionsforthesurnames
list?Atposition0wefind'Rivest',atposition1,'Shamir',andatposition2,
'Adleman'.Ifyouarecuriousaboutwhatthesethreemencreatedtogether,change
print(position,surnames[position])toprint(surnames[position][0],end='')
addafinalprint()outsideoftheloop,andrunthecodeagain.
Now,thisstyleofloopingisactuallymuchclosertolanguageslikeJavaorC++.In
Pythonit’squiteraretoseecodelikethis.Youcanjustiterateoveranysequenceor
collection,sothereisnoneedtogetthelistofpositionsandretrieveelementsoutofa
sequenceateachiteration.It’sexpensive,needlesslyexpensive.Let’schangetheexample
intoamorePythonicform:
simple.for.3.py
surnames=['Rivest','Shamir','Adleman']
forsurnameinsurnames:
print(surname)
Nowthat’ssomething!It’spracticallyEnglish.Theforloopcaniterateoverthesurnames
list,anditgivesbackeachelementinorderateachinteraction.Runningthiscodewill
printthethreesurnames,oneatatime.It’smucheasiertoread,right?
Whatifyouwantedtoprintthepositionaswellthough?Orwhatifyouactuallyneededit
foranyreason?Shouldyougobacktotherange(len(...))form?No.Youcanusethe
enumeratebuilt-infunction,likethis:
simple.for.4.py
surnames=['Rivest','Shamir','Adleman']
forposition,surnameinenumerate(surnames):
print(position,surname)
Thiscodeisveryinterestingaswell.Noticethatenumerategivesbacka2-tuple
(position,surname)ateachiteration,butstill,it’smuchmorereadable(andmore
efficient)thantherange(len(...))example.Youcancallenumeratewithastart
WOW! eBook
www.wowebook.org
parameter,likeenumerate(iterable,start),anditwillstartfromstart,ratherthan0.
Justanotherlittlethingthatshowsyouhowmuchthoughthasbeengivenindesigning
Pythonsothatitmakesyourlifeeasy.
Usingaforloopitispossibletoiterateoverlists,tuples,andingeneralanythingthatin
Pythoniscallediterable.Thisisaveryimportantconcept,solet’stalkaboutitabitmore.
WOW! eBook
www.wowebook.org
Iteratorsanditerables
AccordingtothePythondocumentation,aniterableis:
“Anobjectcapableofreturningitsmembersoneatatime.Examplesofiterables
includeallsequencetypes(suchaslist,str,andtuple)andsomenon-sequence
typeslikedict,fileobjects,andobjectsofanyclassesyoudefinewithan
__iter__()or__getitem__()method.Iterablescanbeusedinaforloopandin
manyotherplaceswhereasequenceisneeded(zip(),map(),…).Whenaniterable
objectispassedasanargumenttothebuilt-infunctioniter(),itreturnsaniterator
fortheobject.Thisiteratorisgoodforonepassoverthesetofvalues.Whenusing
iterables,itisusuallynotnecessarytocalliter()ordealwithiteratorobjects
yourself.Theforstatementdoesthatautomaticallyforyou,creatingatemporary
unnamedvariabletoholdtheiteratorforthedurationoftheloop.”
Simplyput,whathappenswhenyouwriteforkinsequence:...body…,isthatthe
forloopaskssequenceforthenextelement,itgetssomethingback,itcallsthat
somethingk,andthenexecutesitsbody.Then,onceagain,theforloopaskssequence
againforthenextelement,itcallsitkagain,andexecutesthebodyagain,andsoonand
soforth,untilthesequenceisexhausted.Emptysequenceswillresultinzeroexecutionsof
thebody.
Somedatastructures,wheniteratedover,producetheirelementsinorder,likelists,tuples,
andstrings,whilesomeothersdon’t,likesetsanddictionaries.
Pythongivesustheabilitytoiterateoveriterables,usingatypeofobjectcallediterator.
Accordingtotheofficialdocumentation,aniteratoris:
“Anobjectrepresentingastreamofdata.Repeatedcallstotheiterator’s__next__()
method(orpassingittothebuilt-infunctionnext())returnsuccessiveitemsinthe
stream.WhennomoredataareavailableaStopIterationexceptionisraised
instead.Atthispoint,theiteratorobjectisexhaustedandanyfurthercallstoits
__next__()methodjustraiseStopIterationagain.Iteratorsarerequiredtohave
an__iter__()methodthatreturnstheiteratorobjectitselfsoeveryiteratorisalso
iterableandmaybeusedinmostplaceswhereotheriterablesareaccepted.One
notableexceptioniscodewhichattemptsmultipleiterationpasses.Acontainerobject
(suchasalist)producesafreshnewiteratoreachtimeyoupassittotheiter()
functionoruseitinaforloop.Attemptingthiswithaniteratorwilljustreturnthe
sameexhaustediteratorobjectusedinthepreviousiterationpass,makingitappear
likeanemptycontainer.”
Don’tworryifyoudon’tfullyunderstandalltheprecedinglegalese,youwillinduetime.
Iputithereasahandyreferenceforthefuture.
Inpractice,thewholeiterable/iteratormechanismissomewhathiddenbehindthecode.
Unlessyouneedtocodeyourowniterableoriteratorforsomereason,youwon’thaveto
worryaboutthistoomuch.Butit’sveryimportanttounderstandhowPythonhandlesthis
WOW! eBook
www.wowebook.org
keyaspectofcontrolflowbecauseitwillshapethewayyouwillwriteyourcode.
WOW! eBook
www.wowebook.org
Iteratingovermultiplesequences
Let’sseeanotherexampleofhowtoiterateovertwosequencesofthesamelength,in
ordertoworkontheirrespectiveelementsinpairs.Saywehavealistofpeopleandalist
ofnumbersrepresentingtheageofthepeopleinthefirstlist.Wewanttoprintapair
person/ageononelineforallofthem.Let’sstartwithanexampleandlet’srefineit
gradually.
multiple.sequences.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
forpositioninrange(len(people)):
person=people[position]
age=ages[position]
print(person,age)
Bynow,thiscodeshouldbeprettystraightforwardforyoutounderstand.Weneedto
iterateoverthelistofpositions(0,1,2,3)becausewewanttoretrieveelementsfromtwo
differentlists.Executingitwegetthefollowing:
$pythonmultiple.sequences.py
Jonas25
Julio30
Mike31
Mez39
ThiscodeisbothinefficientandnotPythonic.Inefficientbecauseretrievinganelement
giventhepositioncanbeanexpensiveoperation,andwe’redoingitfromscratchateach
iteration.Themailmandoesn’tgobacktothebeginningoftheroadeachtimehedelivers
aletter,right?Hemovesfromhousetohouse.Fromonetothenextone.Let’strytomake
itbetterusingenumerate:
multiple.sequences.enumerate.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
forposition,personinenumerate(people):
age=ages[position]
print(person,age)
Better,butstillnotperfect.Andstillabitugly.We’reiteratingproperlyonpeople,but
we’restillfetchingageusingpositionalindexing,whichwewanttoloseaswell.Well,no
worries,Pythongivesyouthezipfunction,remember?Let’suseit!
multiple.sequences.zip.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
forperson,ageinzip(people,ages):
print(person,age)
Ah!Somuchbetter!Onceagain,comparetheprecedingcodewiththefirstexampleand
admirePython’selegance.ThereasonIwantedtoshowthisexampleistwofold.Onthe
WOW! eBook
www.wowebook.org
onehand,IwantedtogiveyouanideaofhowshorterthecodeinPythoncanbecompared
tootherlanguageswherethesyntaxdoesn’tallowyoutoiterateoversequencesor
collectionsaseasily.Andontheotherhand,andmuchmoreimportantly,noticethatwhen
theforloopaskszip(sequenceA,sequenceB)forthenextelement,itgetsbackatuple,
notjustasingleobject.Itgetsbackatuplewithasmanyelementsasthenumberof
sequenceswefeedtothezipfunction.Let’sexpandalittleonthepreviousexamplein
twoways:usingexplicitandimplicitassignment:
multiple.sequences.explicit.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
nationalities=['Belgium','Spain','England','Bangladesh']
forperson,age,nationalityinzip(people,ages,nationalities):
print(person,age,nationality)
Intheprecedingcode,weaddedthenationalitieslist.Nowthatwefeedthreesequencesto
thezipfunction,theforloopgetsbacka3-tupleateachiteration.Noticethattheposition
oftheelementsinthetuplerespectsthepositionofthesequencesinthezipcall.
Executingthecodewillyieldthefollowingresult:
$pythonmultiple.sequences.explicit.py
Jonas25Belgium
Julio30Spain
Mike31England
Mez39Bangladesh
Sometimes,forreasonsthatmaynotbeclearinasimpleexampleliketheprecedingone,
youmaywanttoexplodethetuplewithinthebodyoftheforloop.Ifthatisyourdesire,
it’sperfectlypossibletodoso.
multiple.sequences.implicit.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
nationalities=['Belgium','Spain','England','Bangladesh']
fordatainzip(people,ages,nationalities):
person,age,nationality=data
print(person,age,nationality)
It’sbasicallydoingwhattheforloopdoesautomaticallyforyou,butinsomecasesyou
maywanttodoityourself.Here,the3-tupledatathatcomesfromzip(...),isexploded
withinthebodyoftheforloopintothreevariables:person,age,andnationality.
WOW! eBook
www.wowebook.org
Thewhileloop
Intheprecedingpages,wesawtheforloopinaction.It’sincrediblyusefulwhenyou
needtoloopoverasequenceoracollection.Thekeypointtokeepinmind,whenyou
needtobeabletodiscriminatewhichloopingconstructtouse,isthattheforlooprocks
whenyouhavetoiterateoverafiniteamountofelements.Itcanbeahugeamount,but
still,somethingthatatsomepointends.
Thereareothercasesthough,whenyoujustneedtoloopuntilsomeconditionissatisfied,
orevenloopindefinitelyuntiltheapplicationisstopped.Caseswherewedon’treallyhave
somethingtoiterateon,andthereforetheforloopwouldbeapoorchoice.Butfearnot,
forthesecasesPythonprovidesuswiththewhileloop.
Thewhileloopissimilartotheforloop,inthattheybothloopandateachiterationthey
executeabodyofinstructions.Whatisdifferentbetweenthemisthatthewhileloop
doesn’tloopoverasequence(itcan,butyouhavetomanuallywritethelogicandit
wouldn’tmakeanysense,youwouldjustwanttouseaforloop),rather,itloopsaslong
asacertainconditionissatisfied.Whentheconditionisnolongersatisfied,theloopends.
Asusual,let’sseeanexamplewhichwillclarifyeverythingforus.Wewanttoprintthe
binaryrepresentationofapositivenumber.Inordertodoso,werepeatedlydividethe
numberbytwo,collectingtheremainder,andthenproducetheinverseofthelistof
remainders.Letmegiveyouasmallexampleusingnumber6,whichis110inbinary.
6/2=3(remainder:0)
3/2=1(remainder:1)
1/2=0(remainder:1)
Listofremainders:0,1,1.
Inverseis1,1,0,whichisalsothebinaryrepresentationof6:110
Let’swritesomecodetocalculatethebinaryrepresentationfornumber39:1001112.
binary.py
n=39
remainders=[]
whilen>0:
remainder=n%2#remainderofdivisionby2
remainders.append(remainder)#wekeeptrackofremainders
n//=2#wedividenby2
#reassignthelisttoitsreversedcopyandprintit
remainders=remainders[::-1]
print(remainders)
Intheprecedingcode,Ihighlightedtwothings:n>0,whichistheconditiontokeep
looping,andremainders[::-1]whichisaniceandeasywaytogetthereversedversion
ofalist(missingstartandendparameters,step=-1,producesthesamelist,fromend
tostart,inreverseorder).Wecanmakethecodealittleshorter(andmorePythonic),by
usingthedivmodfunction,whichiscalledwithanumberandadivisor,andreturnsatuple
withtheresultoftheintegerdivisionanditsremainder.Forexample,divmod(13,5)
WOW! eBook
www.wowebook.org
wouldreturn(2,3),andindeed5*2+3=13.
binary.2.py
n=39
remainders=[]
whilen>0:
n,remainder=divmod(n,2)
remainders.append(remainder)
#reassignthelisttoitsreversedcopyandprintit
remainders=remainders[::-1]
print(remainders)
Intheprecedingcode,wehavereassignedntotheresultofthedivisionby2,andthe
remainder,inonesingleline.
Noticethattheconditioninawhileloopisaconditiontocontinuelooping.Ifitevaluates
toTrue,thenthebodyisexecutedandthenanotherevaluationfollows,andsoon,untilthe
conditionevaluatestoFalse.Whenthathappens,theloopisexitedimmediatelywithout
executingitsbody.
Note
IftheconditionneverevaluatestoFalse,theloopbecomesasocalledinfiniteloop.
Infiniteloopsareusedforexamplewhenpollingfromnetworkdevices:youaskthesocket
ifthereisanydata,youdosomethingwithitifthereisany,thenyousleepforasmall
amountoftime,andthenyouaskthesocketagain,overandoveragain,withoutever
stopping.
Havingtheabilitytoloopoveracondition,ortoloopindefinitely,isthereasonwhythe
forloopaloneisnotenough,andthereforePythonprovidesthewhileloop.
Tip
Bytheway,ifyouneedthebinaryrepresentationofanumber,checkoutthebinfunction.
Justforfun,let’sadaptoneoftheexamples(multiple.sequences.py)usingthewhile
logic.
multiple.sequences.while.py
people=['Jonas','Julio','Mike','Mez']
ages=[25,30,31,39]
position=0
whileposition<len(people):
person=people[position]
age=ages[position]
print(person,age)
position+=1
Intheprecedingcode,Ihavehighlightedtheinitialization,condition,andupdateofthe
variableposition,whichmakesitpossibletosimulatetheequivalentforloopcodeby
handlingtheiterationvariablemanually.Everythingthatcanbedonewithaforloopcan
alsobedonewithawhileloop,eventhoughyoucanseethere’sabitofboilerplateyou
WOW! eBook
www.wowebook.org
havetogothroughinordertoachievethesameresult.Theoppositeisalsotrue,but
simulatinganeverendingwhileloopusingaforlooprequiressomerealtrickery,sowhy
wouldyoudothat?Usetherighttoolforthejob,and99.9%ofthetimesyou’llbefine.
So,torecap,useaforloopwhenyouneedtoiterateoverone(oracombinationof)
iterable,andawhileloopwhenyouneedtoloopaccordingtoaconditionbeingsatisfied
ornot.Ifyoukeepinmindthedifferencebetweenthetwopurposes,youwillnever
choosethewrongloopingconstruct.
Let’snowseehowtoalterthenormalflowofaloop.
WOW! eBook
www.wowebook.org
Thebreakandcontinuestatements
Accordingtothetaskathand,sometimesyouwillneedtoaltertheregularflowofaloop.
Youcaneitherskipasingleiteration(asmanytimesyouwant),oryoucanbreakoutof
theloopentirely.Acommonusecaseforskippingiterationsisforexamplewhenyou’re
iteratingoveralistofitemsandyouneedtoworkoneachofthemonlyifsomecondition
isverified.Ontheotherhand,ifyou’reiteratingoveracollectionofitems,andyouhave
foundoneofthemthatsatisfiessomeneedyouhave,youmaydecidenottocontinuethe
loopentirelyandthereforebreakoutofit.Therearecountlesspossiblescenarios,soit’s
bettertoseeacoupleofexamples.
Let’ssayyouwanttoapplya20%discounttoallproductsinabasketlistforthosewhich
haveanexpirationdateoftoday.Thewayyouachievethisistousethecontinue
statement,whichtellstheloopingconstruct(fororwhile)toimmediatelystopexecution
ofthebodyandgotothenextiteration,ifany.Thisexamplewilltakeusalittledeeper
downtherabbitwhole,sobereadytojump.
discount.py
fromdatetimeimportdate,timedelta
today=date.today()
tomorrow=today+timedelta(days=1)#today+1dayistomorrow
products=[
{'sku':'1','expiration_date':today,'price':100.0},
{'sku':'2','expiration_date':tomorrow,'price':50},
{'sku':'3','expiration_date':today,'price':20},
]
forproductinproducts:
ifproduct['expiration_date']!=today:
continue
product['price']*=0.8#equivalenttoapplying20%discount
print(
'Priceforsku',product['sku'],
'isnow',product['price'])
Youseewestartbyimportingthedateandtimedeltaobjects,thenwesetupour
products.Thosewithsku1and3haveanexpirationdateoftoday,whichmeanswewant
toapply20%discountonthem.Weloopovereachproductandweinspecttheexpiration
date.Ifitisnot(inequalityoperator,!=)today,wedon’twanttoexecutetherestofthe
bodysuite,sowecontinue.
Noticethatisnotimportantwhereinthebodysuiteyouplacethecontinuestatement
(youcanevenuseitmorethanonce).Whenyoureachit,executionstopsandgoesbackto
thenextiteration.Ifwerunthediscount.pymodule,thisistheoutput:
$pythondiscount.py
Priceforsku1isnow80.0
Priceforsku3isnow16.0
Whichshowsyouthatthelasttwolinesofthebodyhaven’tbeenexecutedforskunumber
2.
WOW! eBook
www.wowebook.org
Let’snowseeanexampleofbreakingoutofaloop.Saywewanttotellifatleastanyof
theelementsinalistevaluatestoTruewhenfedtotheboolfunction.Giventhatweneed
toknowifthereisatleastone,whenwefinditwedon’tneedtokeepscanningthelistany
further.InPythoncode,thistranslatestousingthebreakstatement.Let’swritethisdown
intocode:
any.py
items=[0,None,0.0,True,0,7]#Trueand7evaluatetoTrue
found=False#thisiscalled"flag"
foriteminitems:
print('scanningitem',item)
ifitem:
found=True#weupdatetheflag
break
iffound:#weinspecttheflag
print('AtleastoneitemevaluatestoTrue')
else:
print('AllitemsevaluatetoFalse')
Theprecedingcodeissuchacommonpatterninprogramming,youwillseeitalot.When
youinspectitemsthisway,basicallywhatyoudoistosetupaflagvariable,thenstartthe
inspection.Ifyoufindoneelementthatmatchesyourcriteria(inthisexample,that
evaluatestoTrue),thenyouupdatetheflagandstopiterating.Afteriteration,youinspect
theflagandtakeactionaccordingly.Executionyields:
$pythonany.py
scanningitem0
scanningitemNone
scanningitem0.0
scanningitemTrue
AtleastoneitemevaluatestoTrue
SeehowexecutionstoppedafterTruewasfound?
Thebreakstatementactsexactlylikethecontinueone,inthatitstopsexecutingthebody
oftheloopimmediately,butalso,preventsanyotheriterationtorun,effectivelybreaking
outoftheloop.
Thecontinueandbreakstatementscanbeusedtogetherwithnolimitationintheir
number,bothintheforandwhileloopingconstructs.
Tip
Bytheway,thereisnoneedtowritecodetodetectifthereisatleastoneelementina
sequencethatevaluatestoTrue.Justcheckouttheanybuilt-infunction.
WOW! eBook
www.wowebook.org
Aspecialelseclause
OneofthefeaturesI’veseenonlyinthePythonlanguageistheabilitytohaveelse
clausesafterwhileandforloops.It’sveryrarelyused,butit’sdefinitelynicetohave.In
short,youcanhaveanelsesuiteafterafororwhileloop.Iftheloopendsnormally,
becauseofexhaustionoftheiterator(forloop)orbecausetheconditionisfinallynotmet
(whileloop),thentheelsesuite(ifpresent)isexecuted.Incaseexecutionisinterrupted
byabreakstatement,theelseclauseisnotexecuted.Let’stakeanexampleofaforloop
thatiteratesoveragroupofitems,lookingforonethatwouldmatchsomecondition.In
casewedon’tfindatleastonethatsatisfiesthecondition,wewanttoraiseanexception.
Thismeanswewanttoarresttheregularexecutionoftheprogramandsignalthatthere
wasanerror,orexception,thatwecannotdealwith.Exceptionswillbethesubjectof
Chapter7,Testing,Profiling,andDealingwithExceptions,sodon’tworryifyoudon’t
fullyunderstandthemnow.Justbearinmindthattheywillaltertheregularflowofthe
code.Letmenowshowyoutwoexamplesthatdoexactlythesamething,butoneofthem
isusingthespecialfor…elsesyntax.Saythatwewanttofindamongacollectionof
peopleonethatcoulddriveacar.
for.no.else.py
classDriverException(Exception):
pass
people=[('James',17),('Kirk',9),('Lars',13),('Robert',8)]
driver=None
forperson,ageinpeople:
ifage>=18:
driver=(person,age)
break
ifdriverisNone:
raiseDriverException('Drivernotfound.')
Noticetheflagpatternagain.WesetdrivertobeNone,thenifwefindoneweupdatethe
driverflag,andthen,attheendoftheloop,weinspectittoseeifonewasfound.Ikind
ofhavethefeelingthatthosekidswoulddriveaverymetalliccar,butanyway,noticethat
ifadriverisnotfound,aDriverExceptionisraised,signalingtheprogramthatexecution
cannotcontinue(we’relackingthedriver).
Thesamefunctionalitycanberewrittenabitmoreelegantlyusingthefollowingcode:
for.else.py
classDriverException(Exception):
pass
people=[('James',17),('Kirk',9),('Lars',13),('Robert',8)]
forperson,ageinpeople:
ifage>=18:
driver=(person,age)
break
else:
WOW! eBook
www.wowebook.org
raiseDriverException('Drivernotfound.')
Noticethatwearen’tforcedtousetheflagpatternanymore.Theexceptionisraisedas
partoftheforlooplogic,whichmakesgoodsensebecausetheforloopischeckingon
somecondition.Allweneedistosetupadriverobjectincasewefindone,becausethe
restofthecodeisgoingtousethatinformationsomewhere.Noticethecodeisshorterand
moreelegant,becausethelogicisnowcorrectlygroupedtogetherwhereitbelongs.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Puttingthisalltogether
Nowthatyouhaveseenallthereistoseeaboutconditionalsandloops,it’stimetospice
thingsupalittle,andseethosetwoexamplesIanticipatedatthebeginningofthischapter.
We’llmixandmatchhere,soyoucanseehowonecanusealltheseconceptstogether.
Let’sstartbywritingsomecodetogeneratealistofprimenumbersuptosomelimit.
PleasebearinmindthatI’mgoingtowriteaveryinefficientandrudimentaryalgorithmto
detectprimes.Theimportantthingforyouistoconcentrateonthosebitsinthecodethat
belongtothischapter’ssubject.
WOW! eBook
www.wowebook.org
Example1–aprimegenerator
AccordingtoWikipedia:
“Aprimenumber(oraprime)isanaturalnumbergreaterthan1thathasnopositive
divisorsotherthan1anditself.Anaturalnumbergreaterthan1thatisnotaprime
numberiscalledacompositenumber.”
Basedonthisdefinition,ifweconsiderthefirst10naturalnumbers,wecanseethat2,3,
5,and7areprimes,while1,4,6,8,9,10arenot.Inordertohaveacomputertellyouifa
numberNisprime,youcandividethatnumberbyallnaturalnumbersintherange[2,N).
Ifanyofthosedivisionsyieldszeroasaremainder,thenthenumberisnotaprime.
Enoughchatter,let’sgetdowntobusiness.I’llwritetwoversionsofthis,thesecondof
whichwillexploitthefor…elsesyntax.
primes.py
primes=[]#thiswillcontaintheprimesintheend
upto=100#thelimit,inclusive
forninrange(2,upto+1):
is_prime=True#flag,newateachiterationofouterfor
fordivisorinrange(2,n):
ifn%divisor==0:
is_prime=False
break
ifis_prime:#checkonflag
primes.append(n)
print(primes)
Lotsofthingstonoticeintheprecedingcode.Firstofallwesetupanemptylistprimes,
whichwillcontaintheprimesattheend.Thelimitis100,andyoucanseeit’sinclusivein
thewaywecallrange()intheouterloop.Ifwewroterange(2,upto)thatwouldbe[2,
upto),right?Thereforerange(2,upto+1)givesus[2,upto+1)==[2,upto].
So,twoforloops.Intheouteroneweloopoverthecandidateprimes,thatis,allnatural
numbersfrom2toupto.Insideeachiterationofthisouterloopwesetupaflag(whichis
settoTrueateachiteration),andthenstartdividingthecurrentnbyallnumbersfrom2to
n–1.Ifwefindaproperdivisorforn,itmeansniscomposite,andthereforewesetthe
flagtoFalseandbreaktheloop.Noticethatwhenwebreaktheinnerone,theouterone
keepsongoingnormally.Thereasonwhywebreakafterhavingfoundaproperdivisorfor
nisthatwedon’tneedanyfurtherinformationtobeabletotellthatnisnotaprime.
Whenwecheckontheis_primeflag,ifitisstillTrue,itmeanswecouldn’tfindany
numberin[2,n)thatisaproperdivisorforn,thereforenisaprime.Weappendntothe
primeslist,andhop!Anotheriteration,untilnequals100.
Runningthiscodeyields:
$pythonprimes.py
[2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,
71,73,79,83,89,97]
WOW! eBook
www.wowebook.org
Beforeweproceed,onequestion:ofalliterationsoftheouterloop,oneofthemis
differentthanalltheothers.Couldyoutellwhichone,andwhy?Thinkaboutitfora
second,gobacktothecodeandtrytofigureitoutforyourself,andthenkeepreadingon.
Didyoufigureitout?Ifnot,don’tfeelbad,it’sperfectlynormal.Iaskedyoutodoitasa
smallexercisebecauseit’swhatcodersdoallthetime.Theskilltounderstandwhatthe
codedoesbysimplylookingatitissomethingyoubuildovertime.It’sveryimportant,so
trytoexerciseitwheneveryoucan.I’lltellyoutheanswernow:theiterationthatbehaves
differentlyfromallothersisthefirstone.Thereasonisbecauseinthefirstiteration,nis
2.Thereforetheinnermostforloopwon’tevenrun,becauseit’saforloopwhichiterates
overrange(2,2),andwhatisthatifnot[2,2)?Tryitoutforyourself,writeasimplefor
loopwiththatiterable,putaprintinthebodysuite,andseeifanythinghappens(it
won’t…).
Now,fromanalgorithmicpointofviewthiscodeisinefficientsolet’satleastmakeit
morebeautiful:
primes.else.py
primes=[]
upto=100
forninrange(2,upto+1):
fordivisorinrange(2,n):
ifn%divisor==0:
break
else:
primes.append(n)
print(primes)
Muchnicer,right?Theis_primeflagiscompletelygone,andweappendntotheprimes
listwhenweknowtheinnerforloophasn’tencounteredanybreakstatements.Seehow
thecodelookscleanerandreadsbetter?
WOW! eBook
www.wowebook.org
Example2–applyingdiscounts
Inthisexample,IwanttoshowyouatechniqueIlikealot.Inmanyprogramming
languages,otherthantheif/elif/elseconstructs,inwhateverformorsyntaxtheymay
come,youcanfindanotherstatement,usuallycalledswitch/case,thatinPythonis
missing.Itistheequivalentofacascadeofif/elif/…/elif/elseclauses,withasyntax
similartothis(warning!JavaScriptcode!):
switch.js
switch(day_number){
case1:
case2:
case3:
case4:
case5:
day="Weekday";
break;
case6:
day="Saturday";
break;
case0:
day="Sunday";
break;
default:
day="";
alert(day_number+'isnotavaliddaynumber.')
}
Intheprecedingcode,weswitchonavariablecalledday_number.Thismeanswegetits
valueandthenwedecidewhatcaseitfitsin(ifany).From1to5thereisacascade,which
meansnomatterthenumber,[1,5]allgodowntothebitoflogicthatsetsdayas
"Weekday".Thenwehavesinglecasesfor0and6andadefaultcasetopreventerrors,
whichalertsthesystemthatday_numberisnotavaliddaynumber,thatis,notin[0,6].
Pythonisperfectlycapableofrealizingsuchlogicusingif/elif/elsestatements:
switch.py
if1<=day_number<=5:
day='Weekday'
elifday_number==6:
day='Saturday'
elifday_number==0:
day='Sunday'
else:
day=''
raiseValueError(
str(day_number)+'isnotavaliddaynumber.')
Intheprecedingcode,wereproducethesamelogicoftheJavaScriptsnippet,inPython,
usingif/elif/elsestatements.IraisedValueErrorexceptionjustasanexampleatthe
end,ifday_numberisnotin[0,6].Thisisonepossiblewayoftranslatingtheswitch/case
logic,butthereisalsoanotherone,sometimescalleddispatching,whichIwillshowyou
WOW! eBook
www.wowebook.org
inthelastversionofthenextexample.
Tip
Bytheway,didyounoticethefirstlineoftheprevioussnippet?Haveyounoticedthat
Pythoncanmakedouble(actually,evenmultiple)comparisons?It’sjustwonderful!
Let’sstartthenewexamplebysimplywritingsomecodethatassignsadiscountto
customersbasedontheircouponvalue.I’llkeepthelogicdowntoaminimumhere,
rememberthatallwereallycareaboutisconditionalsandloops.
coupons.py
customers=[
dict(id=1,total=200,coupon_code='F20'),#F20:fixed,£20
dict(id=2,total=150,coupon_code='P30'),#P30:percent,30%
dict(id=3,total=100,coupon_code='P50'),#P50:percent,50%
dict(id=4,total=110,coupon_code='F15'),#F15:fixed,£15
]
forcustomerincustomers:
code=customer['coupon_code']
ifcode=='F20':
customer['discount']=20.0
elifcode=='F15':
customer['discount']=15.0
elifcode=='P30':
customer['discount']=customer['total']*0.3
elifcode=='P50':
customer['discount']=customer['total']*0.5
else:
customer['discount']=0.0
forcustomerincustomers:
print(customer['id'],customer['total'],customer['discount'])
Westartbysettingupsomecustomers.Theyhaveanordertotal,acouponcode,andanid.
Imadeupfourdifferenttypesofcoupon,twoarefixedandtwoarepercentagebased.You
canseethatintheif/elif/elsecascadeIapplythediscountaccordingly,andIsetitasa
'discount'keyinthecustomerdict.
AttheendIjustprintoutpartofthedatatoseeifmycodeisworkingproperly.
$pythoncoupons.py
120020.0
215045.0
310050.0
411015.0
Thiscodeissimpletounderstand,butallthoseclausesarekindofclutteringthelogic.It’s
noteasytoseewhat’sgoingonatafirstglance,andIdon’tlikeit.Incaseslikethis,you
canexploitadictionarytoyouradvantage,likethis:
coupons.dict.py
customers=[
dict(id=1,total=200,coupon_code='F20'),#F20:fixed,£20
WOW! eBook
www.wowebook.org
dict(id=2,total=150,coupon_code='P30'),#P30:percent,30%
dict(id=3,total=100,coupon_code='P50'),#P50:percent,50%
dict(id=4,total=110,coupon_code='F15'),#F15:fixed,£15
]
discounts={
'F20':(0.0,20.0),#eachvalueis(percent,fixed)
'P30':(0.3,0.0),
'P50':(0.5,0.0),
'F15':(0.0,15.0),
}
forcustomerincustomers:
code=customer['coupon_code']
percent,fixed=discounts.get(code,(0.0,0.0))
customer['discount']=percent*customer['total']+fixed
forcustomerincustomers:
print(customer['id'],customer['total'],customer['discount'])
Runningtheprecedingcodeyieldsexactlythesameresultwehadfromthesnippetbefore
it.Wesparedtwolines,butmoreimportantly,wegainedalotinreadability,asthebodyof
theforloopnowisjustthreelineslong,andveryeasytounderstand.Theconcepthereis
touseadictionaryasdispatcher.Inotherwords,wetrytofetchsomethingfromthe
dictionarybasedonacode(ourcoupon_code),andbyusingdict.get(key,default),
wemakesurewealsocaterforwhenthecodeisnotinthedictionaryandweneeda
defaultvalue.
NoticethatIhadtoapplysomeverysimplelinearalgebrainordertocalculatethe
discountproperly.Eachdiscounthasapercentageandfixedpartinthedictionary,
representedbya2-tuple.Byapplyingpercent*total+fixed,wegetthecorrect
discount.Whenpercentis0,theformulajustgivesthefixedamount,anditgives
percent*totalwhenfixedis0.Simplebuteffective.
Thistechniqueisimportantbecauseitisalsousedinothercontexts,withfunctions,where
itactuallybecomesmuchmorepowerfulthanwhatwe’veseenintheprecedingsnippet.If
it’snotcompletelycleartoyouhowitworks,Isuggestyoutotakeyourtimeand
experimentwithit.Changevaluesandaddprintstatementstoseewhat’sgoingonwhile
theprogramisrunning.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Aquickpeekattheitertoolsmodule
Achapteraboutiterables,iterators,conditionallogic,andloopingwouldn’tbecomplete
withoutspendingafewwordsabouttheitertoolsmodule.Ifyouareintoiterating,this
isakindofheaven.
AccordingtothePythonofficialdocumentation,theitertoolsmoduleis:
“Amodulewhichimplementsanumberofiteratorbuildingblocksinspiredby
constructsfromAPL,Haskell,andSML.Eachhasbeenrecastinaformsuitablefor
Python.Themodulestandardizesacoresetoffast,memoryefficienttoolsthatare
usefulbythemselvesorincombination.Together,theyforman“iteratoralgebra”
makingitpossibletoconstructspecializedtoolssuccinctlyandefficientlyinpure
Python.”
BynomeansdoIhavetheroomheretoshowyouallthegoodiesyoucanfindinthis
module,soIencourageyoutogoandcheckitoutforyourself,Ipromiseyou’llenjoyit.
Inanutshell,itprovidesyouwiththreebroadcategoriesofiterators.Iwillgiveyouavery
smallexampleofoneiteratortakenfromeachoneofthem,justtomakeyourmouthwater
alittle.
WOW! eBook
www.wowebook.org
Infiniteiterators
Infiniteiteratorsallowyoutoworkwithaforloopinadifferentfashion,likeifitwasa
whileloop.
infinite.py
fromitertoolsimportcount
fornincount(5,3):
ifn>20:
break
print(n,end=',')#insteadofnewline,commaandspace
Runningthecodegivesthis:
$pythoninfinite.py
5,8,11,14,17,20,
Thecountfactoryclassmakesaniteratorthatjustgoesonandoncounting.Itstartsfrom
5andkeepsadding3toit.Weneedtomanuallybreakitifwedon’twanttogetstuckin
aninfiniteloop.
WOW! eBook
www.wowebook.org
Iteratorsterminatingontheshortestinput
sequence
Thiscategoryisveryinteresting.Itallowsyoutocreateaniteratorbasedonmultiple
iterators,combiningtheirvaluesaccordingtosomelogic.Thekeypointhereisthat
amongthoseiterators,incaseanyofthemareshorterthantherest,theresultingiterator
won’tbreak,itwillsimplystopassoonastheshortestiteratorisexhausted.Thisisvery
theoretical,Iknow,soletmegiveyouanexampleusingcompress.Thisiteratorgivesyou
backthedataaccordingtoacorrespondingiteminaselectorbeingTrueorFalse:
compress('ABC',(1,0,1))wouldgiveback'A'and'C',becausetheycorrespondto
the1's.Let’sseeasimpleexample:
compress.py
fromitertoolsimportcompress
data=range(10)
even_selector=[1,0]*10
odd_selector=[0,1]*10
even_numbers=list(compress(data,even_selector))
odd_numbers=list(compress(data,odd_selector))
print(odd_selector)
print(list(data))
print(even_numbers)
print(odd_numbers)
Noticethatodd_selectorandeven_selectorare20elementslong,whiledataisjust10
elementslong.compresswillstopassoonasdatahasyieldeditslastelement.Running
thiscodeproducesthefollowing:
$pythoncompress.py
[0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1]
[0,1,2,3,4,5,6,7,8,9]
[0,2,4,6,8]
[1,3,5,7,9]
It’saveryfastandnicewayofselectingelementsoutofaniterable.Thecodeisvery
simple,justnoticethatinsteadofusingaforlooptoiterateovereachvaluethatisgiven
backbythecompresscalls,weusedlist(),whichdoesthesame,butinsteadof
executingabodyofinstructions,putsallthevaluesintoalistandreturnsit.
WOW! eBook
www.wowebook.org
Combinatoricgenerators
Lastbutnotleast,combinatoricgenerators.Thesearereallyfun,ifyouareintothiskind
ofthing.Let’sjustseeasimpleexampleonpermutations.
AccordingtoWolframMathworld:
“Apermutation,alsocalledan“arrangementnumber”or“order”,isa
rearrangementoftheelementsofanorderedlistSintoaone-to-onecorrespondence
withSitself.”
Forexample,thepermutationsofABCare6:ABC,ACB,BAC,BCA,CAB,andCBA.
IfasethasNelements,thenthenumberofpermutationsofthemisN!(Nfactorial).For
thestringABCthepermutationsare3!=3*2*1=6.Let’sdoitinPython:
permutations.py
fromitertoolsimportpermutations
print(list(permutations('ABC')))
Thisveryshortsnippetofcodeproducesthefollowingresult:
$pythonpermutations.py
[('A','B','C'),('A','C','B'),('B','A','C'),('B','C','A'),('C',
'A','B'),('C','B','A')]
Beverycarefulwhenyouplaywithpermutation.Theirnumbergrowsataratethatis
proportionaltothefactorialofthenumberoftheelementsyou’repermuting,andthat
numbercangetreallybig,reallyfast.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,we’vetakenanotherstepforwardtoexpandourcodingvocabulary.We’ve
seenhowtodrivetheexecutionofthecodebyevaluatingconditions,andwe’veseenhow
toloopanditerateoversequencesandcollectionsofobjects.Thisgivesusthepowerto
controlwhathappenswhenourcodeisrun,whichmeanswearegettinganideaonhowto
shapeitsothatitdoeswhatwewantanditreactstodatathatchangesdynamically.
We’vealsoseenhowtocombineeverythingtogetherinacoupleofsimpleexamples,and
intheendwehavetakenabrieflookattheitertoolsmodule,whichisfullofinteresting
iteratorswhichcanenrichourabilitieswithPythonevenmore.
Nowit’stimetoswitchgears,totakeanotherstepforwardandtalkaboutfunctions.The
nextchapterisallaboutthembecausetheyareextremelyimportant.Makesureyou’re
comfortablewithwhathasbeendoneuptonow:Iwanttoprovideyouwithinteresting
examples,soI’llhavetogoalittlefaster.Ready?Turnthepage.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter4.Functions,theBuildingBlocks
ofCode
“Tocreatearchitectureistoputinorder.Putwhatinorder?Functionandobjects.” —LeCorbusier
Inthischapter,we’regoingtoexplorefunctions.Wealreadysaidthateverythingisan
objectinPython,andfunctionsarenoexceptiontothis.But,whatexactlyisafunction?A
functionisasequenceofinstructionsthatperformatask,bundledasaunit.Thisunitcan
thenbeimportedandusedwhereverit’sneeded.Therearemanyadvantagestousing
functionsinyourcode,aswe’llseeshortly.
Ibelievethesaying,apictureisworthonethousandwords,isparticularlytruewhen
explainingfunctionstosomeonewhoisnewtothisconcept,sopleasetakealookatthe
followingimage:
Asyoucansee,afunctionisablockofinstructions,packagedasawhole,likeabox.
Functionscanacceptinputargumentsandproduceoutputvalues.Bothoftheseare
optional,aswe’llseeintheexamplesinthischapter.
AfunctioninPythonisdefinedbyusingthedefkeyword,afterwhichthenameofthe
functionfollows,terminatedbyapairofbraces(whichmayormaynotcontaininput
parameters)and,finally,acolon(:)signalstheendofthefunctiondefinitionline.
Immediatelyafterwards,indentedbyfourspaces,wefindthebodyofthefunction,which
isthesetofinstructionsthatthefunctionwillexecutewhencalled.
Note
Notethattheindentationbyfourspacesisnotmandatory,butitistheamountofspaces
suggestedbyPEP8,and,inpractice,itisthemostwidelyusedspacingmeasure.
Afunctionmayormaynotreturnoutput.Ifafunctionwantstoreturnoutput,itdoessoby
usingthereturnkeyword,followedbythedesiredoutput.Ifyouhaveaneagleeye,you
mayhavenoticedthelittle*afterOptionalintheoutputsectionoftheprecedingpicture.
ThisisbecauseafunctionalwaysreturnssomethinginPython,evenifyoudon’texplicitly
usethereturnclause.Ifthefunctionhasnoreturnstatementinitsbody,it’sreturnvalue
WOW! eBook
www.wowebook.org
isNone.Thereasonsbehindthisdesignchoiceareoutofthescopeofanintroductory
chapter,soallyouneedtoknowisthatthisbehaviorwillmakeyourlifeeasier,asalways,
thankyouPython.
WOW! eBook
www.wowebook.org
Whyusefunctions?
Functionsareamongthemostimportantconceptsandconstructsofanylanguage,solet
megiveyouafewreasonswhyweneedthem:
Theyreducecodeduplicationinaprogram.Byhavingaspecifictasktakencareof
byaniceblockofpackagedcodethatwecanimportandcallwheneverwewant,we
don’tneedtoduplicateitsimplementation.
Theyhelpinsplittingacomplextaskorprocedureintosmallerblocks,eachofwhich
becomesafunction.
Theyhidetheimplementationdetailsfromtheirusers.
Theyimprovetraceability.
Theyimprovereadability.
Let’slookatafewexamplestogetabetterunderstandingofeachpoint.
WOW! eBook
www.wowebook.org
Reducecodeduplication
Imaginethatyouarewritingapieceofscientificsoftware,andyouneedtocalculate
primesuptoalimit,aswedidinthepreviouschapter.Youwriteseveralalgorithmsand
primenumbers,beingthebasisofmanydifferenttypesofcalculations,keepcreepinginto
yourcode.Well,youhaveanicealgorithmtocalculatethem,soyoucopyandpasteitto
whereveryouneed.Oneday,though,yourfriendMisterSmartygivesyouabetter
algorithmtocalculateprimenumbers,andthiswillsaveyoualotoftime.Atthispoint,
youneedtogooveryourwholecodebaseandreplacetheoldcodewiththenewcode.
Thisisactuallyaverybadwaytogoaboutit.It’serror-prone,youneverknowwhatlines
youarechoppingoutorleavingtherebymistakewhenyoucutandpastecodeinother
code,andyoumayalsoriskmissingoneoftheplaceswhereprimecalculationwasdone,
leavingyoursoftwarewithdifferentversions.Canyouimagineifyoudiscoveredthatthe
oldwaywasbuggy?Youwouldhaveanundetectedbuginyourcode,andbugslikethis
arequitehardtospot,especiallyinbigcodebases.
So,whatshouldyoudo?Simple!Youwriteafunction,get_prime_numbers(upto),and
useitanywhereyouneedalistofprimes.WhenMisterSmartycomestoyouandgives
youthenewcode,allyouhavetodoisreplacethebodyofthatfunctionwiththenew
implementation,andyou’redone!Therestofthesoftwarewillautomaticallyadapt,since
it’sjustcallingthefunction.
Yourcodewillbeshorter,itwillnotsufferfrominconsistenciesbetweenoldandnew
waysofperformingatask,orundetectedbugsduetocopyandpastefailuresoroversights.
Usefunctions,andyou’llonlygainfromit,Ipromise.
WOW! eBook
www.wowebook.org
Splittingacomplextask
Functionsareveryusefulalsotosplitalongorcomplextaskintosmallerpieces.Theend
resultisthatthecodebenefitsfromitinseveralways,forexample,readability,testability,
andreuse.Togiveyouasimpleexample,imaginethatyou’repreparingareport.Your
codeneedstofetchdatafromadatasource,parseit,filterit,polishit,andthenawhole
seriesofalgorithmsneedstoberunagainstit,inordertoproducetheresultswhichwill
feedtheReportclass.It’snotuncommontoreadprocedureslikethisthatarejustonebig
functiondo_report(data_source).Therearetensorhundredsoflinesofcodewhichend
withreturnreport.
Situationslikethisarecommonincodeproducedbyscientists.Theyhavebrilliantminds
andtheycareaboutthecorrectnessoftheendresultbut,unfortunately,sometimesthey
havenotraininginprogrammingtheory.Itisnottheirfault,onecannotknoweverything.
Now,pictureinyourheadsomethinglikeafewhundredlinesofcode.It’sveryhardto
followthrough,tofindtheplaceswherethingsarechangingcontext(likefinishingone
taskandstartingthenextone).Doyouhavethepictureinyourmind?Good.Don’tdoit!
Instead,lookatthiscode:
data.science.example.py
defdo_report(data_source):
#fetchandpreparedata
data=fetch_data(data_source)
parsed_data=parse_data(data)
filtered_data=filter_data(parsed_data)
polished_data=polish_data(filtered_data)
#runalgorithmsondata
final_data=analyse(polished_data)
#createandreturnreport
report=Report(final_data)
returnreport
Thepreviousexampleisfictitious,ofcourse,butcanyouseehoweasyitwouldbetogo
throughthecode?Iftheendresultlookswrong,itwouldbeveryeasytodebugeachofthe
singledataoutputsinthedo_reportfunction.Moreover,it’seveneasiertoexcludepartof
theprocesstemporarilyfromthewholeprocedure(youjustneedtocommentouttheparts
youneedtosuspend).Codelikethisiseasiertodealwith.
WOW! eBook
www.wowebook.org
Hideimplementationdetails
Let’sstaywiththeprecedingexampletotalkaboutthispointaswell.Youcanseethat,by
goingthroughthecodeofthedo_reportfunction,youcangetaprettygood
understandingwithoutreadingonesinglelineofimplementation.Thisisbecause
functionshidetheimplementationdetails.Thisfeaturemeansthat,ifyoudon’tneedto
delveintodetails,youarenotforcedto,inthewayyouwouldifdo_reportwasjustone
bigfatfunction.Inordertounderstandwhatwasgoingon,youwouldhavetoreadthe
implementationdetails.Youdon’tneedtowithfunctions.Thisreducesthetimeyouspend
readingthecodeandsince,inaprofessionalenvironment,readingcodetakesmuchmore
timethanactuallywritingit,it’sveryimportanttoreduceitasmuchaswecan.
WOW! eBook
www.wowebook.org
Improvereadability
Coderssometimesdon’tseethepointinwritingafunctionwithabodyofoneortwolines
ofcode,solet’slookatanexamplethatshowsyouwhyyoushoulddoit.
Imaginethatyouneedtomultiplytwomatrices:
Wouldyouprefertohavetoreadthiscode:
matrix.multiplication.nofunc.py
a=[[1,2],[3,4]]
b=[[5,1],[2,1]]
c=[[sum(i*jfori,jinzip(r,c))forcinzip(*b)]
forrina]
Orwouldyoupreferthisone:
matrix.multiplication.func.py
#thisfunctioncouldalsobedefinedinanothermodule
defmatrix_mul(a,b):
return[[sum(i*jfori,jinzip(r,c))forcinzip(*b)]
forrina]
a=[[1,2],[3,4]]
b=[[5,1],[2,1]]
c=matrix_mul(a,b)
It’smucheasiertounderstandthatcistheresultofthemultiplicationbetweenaandbin
thesecondexample.It’smucheasiertoreadthroughthecodeand,ifyoudon’tneedto
modifythatpart,youdon’tevenneedtogointotheimplementationdetails.
Therefore,readabilityisimprovedherewhile,inthefirstsnippet,youwouldhaveto
spendtimetryingtounderstandwhatthatcomplicatedlistcomprehensionwasdoing.
Note
Don’tworryifyoudon’tunderstandlistcomprehensions,we’llstudytheminthenext
chapter.
WOW! eBook
www.wowebook.org
Improvetraceability
Imaginethatyouhavewrittenane-commercewebsite.Youhavedisplayedtheproduct
pricesalloverthepages.ImaginethatthepricesinyourdatabasearestoredwithnoVAT,
butyouwanttodisplaythemonthewebsitewithVATat20%.Here’safewwaysof
calculatingtheVAT-inclusivepricefromtheVAT-exclusiveprice.
vat.py
price=100#GBP,noVAT
final_price1=price*1.2
final_price2=price+price/5.0
final_price3=price*(100+20)/100.0
final_price4=price+price*0.2
AllthesefourdifferentwaysofcalculatingaVAT-inclusivepriceareperfectlyacceptable,
andIpromiseyouIhavefoundthemallinmycolleagues’code,overtheyears.Now,
imaginethatyouhavestartedsellingyourproductsindifferentcountriesandsomeof
themhavedifferentVATratessoyouneedtorefactoryourcode(throughoutthewebsite)
inordertomakethatVATcalculationdynamic.
HowdoyoutracealltheplacesinwhichyouareperformingaVATcalculation?Coding
todayisacollaborativetaskandyoucannotbesuretheVAThasbeencalculatedusing
onlyoneofthoseforms.It’sgoingtobehell,believeme.
So,let’swriteafunctionthattakestheinputvalues,vatandprice(VAT-exclusive),and
returnsaVAT-inclusiveprice.
vat.function.py
defcalculate_price_with_vat(price,vat):
returnprice*(100+vat)/100
Nowyoucanimportthatfunctionandapplyitinanyplaceofyourwebsitewhereyou
needtocalculateaVAT-inclusivepriceandwhenyouneedtotracethosecalls,youcan
searchforcalculate_price_with_vat.
Note
Notethat,intheprecedingexample,priceisassumedtobeVAT-exclusive,andvathasa
percentagevalue(forexample,19,20,23,andsoon).
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Scopesandnameresolution
Doyourememberwhenwetalkedaboutscopesandnamespacesinthefirstchapter?
We’regoingtoexpandonthatconceptnow.Finally,wecantalkaboutfunctionsandthis
willmakeeverythingeasiertounderstand.Let’sstartwithaverysimpleexample.
scoping.level.1.py
defmy_function():
test=1#thisisdefinedinthelocalscopeofthefunction
print('my_function:',test)
test=0#thisisdefinedintheglobalscope
my_function()
print('global:',test)
Ihavedefinedthenametestintwodifferentplacesinthepreviousexample.Itisactually
intwodifferentscopes.Oneistheglobalscope(test=0),andtheotheristhelocal
scopeofthefunctionmy_function(test=1).Ifyouexecutethecode,you’llseethis:
$pythonscoping.level.1.py
my_function:1
global:0
It’sclearthattest=1shadowstheassignmenttest=0inmy_function.Intheglobal
context,testisstill0,asyoucanseefromtheoutputoftheprogrambutwedefinethe
nametestagaininthefunctionbody,andwesetittopointtoanintegerofvalue1.Both
thetwotestnamesthereforeexist,oneintheglobalscope,pointingtoanintobjectwith
value0,theotherinthemy_functionscope,pointingtoanintobjectwithvalue1.Let’s
commentoutthelinewithtest=1.Pythongoesandsearchesforthenametestinthe
nextenclosingnamespace(recalltheLEGBrule:Local,Enclosing,Global,Built-in
describedinChapter1,IntroductionandFirstSteps–TakeaDeepBreath)and,inthis
case,wewillseethevalue0printedtwice.Tryitinyourcode.
Now,let’sraisethestakeshereandlevelup:
scoping.level.2.py
defouter():
test=1#outerscope
definner():
test=2#innerscope
print('inner:',test)
inner()
print('outer:',test)
test=0#globalscope
outer()
print('global:',test)
Intheprecedingcode,wehavetwolevelsofshadowing.Onelevelisinthefunction
outer,andtheotheroneisinthefunctioninner.Itisfarfromrocketscience,butitcan
betricky.Ifwerunthecode,weget:
WOW! eBook
www.wowebook.org
$pythonscoping.level.2.py
inner:2
outer:1
global:0
Trycommentingoutthelinetest=1.Whatdoyouthinktheresultwillbe?Well,when
reachingthelineprint('outer:',test),Pythonwillhavetolookfortestinthenext
enclosingscope,thereforeitwillfindandprint0,insteadof1.Makesureyoucomment
outtest=2aswell,toseeifyouunderstandwhathappens,andiftheLEGBruleis
clear,beforeproceeding.
AnotherthingtonoteisthatPythongivesyoutheabilitytodefineafunctioninanother
function.Theinnerfunction’snameisdefinedwithinthenamespaceoftheouterfunction,
exactlyaswouldhappenwithanyothername.
WOW! eBook
www.wowebook.org
Theglobalandnonlocalstatements
Goingbacktotheprecedingexample,wecanalterwhathappenstotheshadowingofthe
testnamebyusingoneofthesetwospecialstatements:globalandnonlocal.Asyoucan
seefromthepreviousexample,whenwedefinetest=2inthefunctioninner,we
overwritetestneitherinthefunctionouter,norintheglobalscope.Wecangetread
accesstothosenamesifweusetheminanestedscopethatdoesn’tdefinethem,butwe
cannotmodifythembecause,whenwewriteanassignmentinstruction,we’reactually
defininganewnameinthecurrentscope.
Howdowechangethisbehavior?Well,wecanusethenonlocalstatement.Accordingto
theofficialdocumentation:
“Thenonlocalstatementcausesthelistedidentifierstorefertopreviouslybound
variablesinthenearestenclosingscopeexcludingglobals.”
Let’sintroduceitinthefunctioninner,andseewhathappens:
scoping.level.2.nonlocal.py
defouter():
test=1#outerscope
definner():
nonlocaltest
test=2#nearestenclosingscope
print('inner:',test)
inner()
print('outer:',test)
test=0#globalscope
outer()
print('global:',test)
NoticehowinthebodyofthefunctioninnerIhavedeclaredthetestnametobe
nonlocal.Runningthiscodeproducesthefollowingresult:
$pythonscoping.level.2.nonlocal.py
inner:2
outer:2
global:0
Wow,lookatthatresult!Itmeansthat,bydeclaringtesttobenonlocalinthefunction
inner,weactuallygettobindthenametesttothatdeclaredinthefunctionouter.Ifwe
removedthenonlocaltestlinefromthefunctioninnerandtriedthesametrickinthe
functionouter,wewouldgetaSyntaxError,becausethenonlocalstatementworkson
enclosingscopesexcludingtheglobalone.
Isthereawaytogettothattest=0intheglobalnamespacethen?Ofcourse,wejust
needtousetheglobalstatement.Let’stryit.
scoping.level.2.global.py
WOW! eBook
www.wowebook.org
defouter():
test=1#outerscope
definner():
globaltest
test=2#globalscope
print('inner:',test)
inner()
print('outer:',test)
test=0#globalscope
outer()
print('global:',test)
Notethatwehavenowdeclaredthenametesttobeglobal,whichwillbasicallybindit
totheonewedefinedintheglobalnamespace(test=0).Runthecodeandyoushould
getthefollowing:
$pythonscoping.level.2.global.py
inner:2
outer:1
global:2
Thisshowsthatthenameaffectedbytheassignmenttest=2isnowtheglobalone.
Thistrickwouldalsoworkintheouterfunctionbecause,inthiscase,we’rereferringto
theglobalscope.Tryitforyourselfandseewhatchanges,getcomfortablewithscopes
andnameresolution,it’sveryimportant.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Inputparameters
Atthebeginningofthischapter,wesawthatafunctioncantakeinputparameters.Before
wedelveintoallpossibletypeofparameters,let’smakesureyouhaveaclear
understandingofwhatpassingaparametertoafunctionmeans.Therearethreekeypoints
tokeepinmind:
Argumentpassingisnothingmorethanassigninganobjecttoalocalvariablename
Assigninganobjecttoanargumentnameinsideafunctiondoesn’taffectthecaller
Changingamutableobjectargumentinafunctionaffectsthecaller
Let’slookatanexampleforeachofthesepoints.
WOW! eBook
www.wowebook.org
Argumentpassing
Takealookatthefollowingcode.Wedeclareanamexintheglobalscope,thenwe
declareafunctionfunc(y)andwecallit,passingx.Ihighlightedthecallinthecode.
key.points.argument.passing.py
x=3
deffunc(y):
print(y)
func(x)#prints:3
Whenfunciscalledwithx,whathappensisthatwithinitslocalscope,anameyis
created,andit’spointedtothesameobjectxispointingto.Thisisbetterclarifiedbythe
followingpicture:
Therightpartoftheprecedingpicturedepictsthestateoftheprogramwhenexecutionhas
reachedtheend,afterfunchasreturned(None).TakealookattheFramescolumn,and
notethatwehavetwonames,xandfunc,intheglobalnamespace(Globalframe),
pointingtoanint(withavalueofthree)andtoafunctionobject,respectively.Right
belowit,intherectangletitledfunc,wecanseethefunction’slocalnamespace,inwhich
onlyonenamehasbeendefined:y.Becausewehavecalledfuncwithx(line5intheleft
partofthepicture),yispointingtothesameobjectthatxispointingto.Thisiswhat
happensunderthehoodwhenanargumentispassedtoafunction.Ifwehadusedthe
namexinsteadofyinthefunctiondefinition,thingswouldhavebeenexactlythesame
(onlymaybeabitconfusingatfirst),therewouldbealocalxinthefunction,andaglobal
xoutside,aswesawintheScopesandnameresolutionsection.
So,inanutshell,whatreallyhappensisthatthefunctioncreatesinitslocalscopethe
namesdefinedasargumentsand,whenwecallit,webasicallytellPythonwhichobjects
thosenamesmustbepointedtowards.
WOW! eBook
www.wowebook.org
Assignmenttoargumentnamesdon’taffectthe
caller
Thisissomethingthatcanbetrickytounderstandatfirst,solet’slookatanexample.
key.points.assignment.py
x=3
deffunc(x):
x=7#definingalocalx,notchangingtheglobalone
func(x)
print(x)#prints:3
Intheprecedingcode,whenthelinex=7isexecuted,whathappensisthatwithinthe
localscopeofthefunctionfunc,thenamexispointedtoanintegerwithvalue7,leaving
theglobalxunaltered.
WOW! eBook
www.wowebook.org
Changingamutableaffectsthecaller
Thisisthefinalpoint,andit’sveryimportantbecausePythonapparentlybehaves
differentlywithmutables(justapparentlythough).Let’slookatanexample:
key.points.mutable.py
x=[1,2,3]
deffunc(x):
x[1]=42#thisaffectsthecaller!
func(x)
print(x)#prints:[1,42,3]
Wow,weactuallychangedtheoriginalobject!Ifyouthinkaboutit,thereisnothingweird
inthisbehavior.Thenamexinthefunctionissettopointtothecallerobjectbythe
functioncallandwithinthebodyofthefunction,we’renotchangingx,inthatwe’renot
changingitsreference,or,inotherwords,wearenotchangingtheobjectxispointingto.
Whatwe’redoingisaccessingthatobject’selementatposition1,andchangingitsvalue.
Rememberpoint#2:“Assigninganobjecttoanargumentnamewithinafunctiondoesn’t
affectthecaller“.Ifthatiscleartoyou,thefollowingcodeshouldnotbesurprising.
key.points.mutable.assignment.py
x=[1,2,3]
deffunc(x):
x[1]=42#thischangesthecaller!
x='somethingelse'#thispointsxtoanewstringobject
func(x)
print(x)#stillprints:[1,42,3]
TakealookatthetwolinesIhavehighlighted.Atfirst,wejustaccessthecallerobject
again,atposition1,andchangeitsvaluetonumber42.Then,wereassignxtopointtothe
string'somethingelse'.Thisleavesthecallerunaltered,accordingtopoint#2,and,in
fact,theoutputisthesameasthatoftheprevioussnippet.
Takeyourtimetoplayaroundwiththisconceptandexperimentwithprintsandcallsto
theidfunctionuntileverythingisclearinyourmind.Thisisoneofthekeyaspectsof
Pythonanditmustbeveryclear,otherwiseyouriskintroducingsubtlebugsintoyour
code.
Nowthatwehaveagoodunderstandingofinputparametersandhowtheybehave,let’s
seehowwecanspecifythem.
WOW! eBook
www.wowebook.org
Howtospecifyinputparameters
Therearefivedifferentwaysofspecifyinginputparameters.Let’slookatthemoneby
one.
Positionalarguments
Positionalargumentsarereadfromlefttorightandtheyarethemostcommontypeof
arguments.
arguments.positional.py
deffunc(a,b,c):
print(a,b,c)
func(1,2,3)#prints:123
Thereisnotmuchelsetosay.Theycanbeasnumerousasyouwantandtheyareassigned
byposition.Inthefunctioncall,1comesfirst,2comessecondand3comesthird,
thereforetheyareassignedtoa,bandcrespectively.
Keywordargumentsanddefaultvalues
Keywordargumentsareassignedbykeywordusingthename=valuesyntax.
arguments.keyword.py
deffunc(a,b,c):
print(a,b,c)
func(a=1,c=2,b=3)#prints:132
Keywordargumentsactwhencallingthefunctioninsteadofrespectingtheleft-to-right
positionalassignment,k.Keywordargumentsarematchedbyname,evenwhentheydon’t
respectthedefinition’soriginalposition(we’llseethatthereisalimitationtothisbehavior
later,whenwemixandmatchdifferenttypesofarguments).
Thecounterpartofkeywordarguments,onthedefinitionside,isdefaultvalues.The
syntaxisthesame,name=value,andallowsustonothavetoprovideanargumentifwe
arehappywiththegivendefault.
arguments.default.py
deffunc(a,b=4,c=88):
print(a,b,c)
func(1)#prints:1488
func(b=5,a=7,c=9)#prints:759
func(42,c=9)#prints:4249
Thearetwothingstonotice,whichareveryimportant.Firstofall,youcannotspecifya
defaultargumentontheleftofapositionalone.Second,notehowintheexamples,when
anargumentispassedwithoutusingtheargument_name=valuesyntax,itmustbethefirst
oneinthelist,,anditisalwaysassignedtoa.Tryandscramblethoseargumentsandsee
whathappens.Pythonerrormessagesareverygoodattellingyouwhat’swrong.So,for
example,ifyoutriedsomethinglikethis:
WOW! eBook
www.wowebook.org
func(b=1,c=2,42)#positionalargumentafterkeywordone
Youwouldgetthefollowingerror:
SyntaxError:non-keywordargafterkeywordarg
Thisinformsyouthatyou’vecalledthefunctionincorrectly.
Variablepositionalarguments
Sometimesyoumaywanttopassavariablenumberofpositionalargumentstoafunction
andPythonprovidesyouwiththeabilitytodoit.Let’slookataverycommonusecase,
theminimumfunction.Thisisafunctionthatcalculatestheminimumofitsinputvalues.
arguments.variable.positional.py
defminimum(*n):
#print(n)#nisatuple
ifn:#explainedafterthecode
mn=n[0]
forvalueinn[1:]:
ifvalue<mn:
mn=value
print(mn)
minimum(1,3,-7,9)#n=(1,3,-7,9)-prints:-7
minimum()#n=()-prints:nothing
Asyoucansee,whenwespecifyaparameterprependinga*toitsname,wearetelling
Pythonthatthatparameterwillbecollectingavariablenumberofpositionalarguments,
accordingtohowthefunctioniscalled.Withinthefunction,nisatuple.Uncommentthe
print(n)toseeforyourselfandplayaroundwithitforabit.
Note
Haveyounoticedhowwecheckedifnwasn’temptywithasimpleifn:?Thisisdueto
thefactthatcollectionobjectsevaluatetoTruewhennon-empty,andotherwiseFalsein
Python.Thisistruefortuples,sets,lists,dictionaries,andsoon.
Oneotherthingtonoteisthatwemaywanttothrowanerrorwhenwecallthefunction
withnoarguments,insteadofsilentlydoingnothing.Inthiscontext,we’renotconcerned
aboutmakingthisfunctionrobust,butinunderstandingvariablepositionalarguments.
Let’smakeanotherexampletoshowyoutwothingsthat,inmyexperience,areconfusing
tothosewhoarenewtothis.
arguments.variable.positional.unpacking.py
deffunc(*args):
print(args)
values=(1,3,-7,9)
func(values)#equivalentto:func((1,3,-7,9))
func(*values)#equivalentto:func(1,3,-7,9)
Takeagoodlookatthelasttwolinesoftheprecedingexample.Inthefirstone,wecall
WOW! eBook
www.wowebook.org
funcwithoneargument,afourelementstuple.Inthesecondexample,byusingthe*
syntax,we’redoingsomethingcalledunpacking,whichmeansthatthefourelements
tupleisunpacked,andthefunctioniscalledwithfourarguments:1,3,-7,9.
ThisbehaviorispartofthemagicPythondoestoallowyoutodoamazingthingswhen
callingfunctionsdynamically.
Variablekeywordarguments
Variablekeywordargumentsareverysimilartovariablepositionalarguments.Theonly
differenceisthesyntax(**insteadof*)andthattheyarecollectedinadictionary.
Collectionandunpackingworkinthesameway,solet’slookatanexample:
arguments.variable.keyword.py
deffunc(**kwargs):
print(kwargs)
#Allcallsequivalent.Theyprint:{'a':1,'b':42}
func(a=1,b=42)
func(**{'a':1,'b':42})
func(**dict(a=1,b=42))
Allthecallsareequivalentintheprecedingexample.Youcanseethataddinga**infront
oftheparameternameinthefunctiondefinitiontellsPythontousethatnametocollecta
variablenumberofkeywordparameters.Ontheotherhand,whenwecallthefunction,we
caneitherpassname=valueargumentsexplicitly,orunpackadictionaryusingthesame**
syntax.
Thereasonwhybeingabletopassavariablenumberofkeywordparametersisso
importantmaynotbeevidentatthemoment,so,howaboutamorerealisticexample?
Let’sdefineafunctionthatconnectstoadatabase.Wewanttoconnecttoadefault
databasebysimplycallingthisfunctionwithnoparameters.Wealsowanttoconnectto
anyotherdatabasebypassingthefunctiontheappropriatearguments.Beforeyoureadon,
spendacoupleofminutesfiguringoutasolutionbyyourself.
arguments.variable.db.py
defconnect(**options):
conn_params={
'host':options.get('host','127.0.0.1'),
'port':options.get('port',5432),
'user':options.get('user',''),
'pwd':options.get('pwd',''),
}
print(conn_params)
#wethenconnecttothedb(commentedout)
#db.connect(**conn_params)
connect()
connect(host='127.0.0.42',port=5433)
connect(port=5431,user='fab',pwd='gandalf')
Noteinthefunctionwecanprepareadictionaryofconnectionparameters(conn_params)
inthefunctionusingdefaultvaluesasfallback,allowingthemtobeoverwritteniftheyare
WOW! eBook
www.wowebook.org
providedinthefunctioncall.Therearebetterwaystodothiswithfewerlinesofcodebut
we’renotconcernedwiththatnow.Runningtheprecedingcodeyieldsthefollowing
result:
$pythonarguments.variable.db.py
{'host':'127.0.0.1','pwd':'','user':'','port':5432}
{'host':'127.0.0.42','pwd':'','user':'','port':5433}
{'host':'127.0.0.1','pwd':'gandalf','user':'fab','port':5431}
Notethecorrespondencebetweenthefunctioncallsandtheoutput.Notehowdefault
valuesareeitherthereoroverridden,accordingtowhatwaspassedtothefunction.
Keyword-onlyarguments
Python3allowsforanewtypeofparameter:thekeyword-onlyparameter.Wearegoing
tostudythemonlybrieflyastheirusecasesarenotthatfrequent.Therearetwowaysof
specifyingthem,eitherafterthevariablepositionalarguments,orafterabare*.Let’ssee
anexampleofboth.
arguments.keyword.only.py
defkwo(*a,c):
print(a,c)
kwo(1,2,3,c=7)#prints:(1,2,3)7
kwo(c=4)#prints:()4
#kwo(1,2)#breaks,invalidsyntax,withthefollowingerror
#TypeError:kwo()missing1requiredkeyword-onlyargument:'c'
defkwo2(a,b=42,*,c):
print(a,b,c)
kwo2(3,b=7,c=99)#prints:3799
kwo2(3,c=13)#prints:34213
#kwo2(3,23)#breaks,invalidsyntax,withthefollowingerror
#TypeError:kwo2()missing1requiredkeyword-onlyargument:'c'
Asanticipated,thefunction,kwo,takesavariablenumberofpositionalarguments(a)and
akeyword-onlyfunction,c.Theresultsofthecallsarestraightforwardandyoucan
uncommentthethirdcalltoseewhaterrorPythonreturns.
Thesameappliestothefunction,kwo2,whichdiffersfromkwointhatittakesapositional
argumenta,akeywordargumentb,andthenakeyword-onlyargument,c.Youcan
uncommentthethirdcalltoseetheerror.
Nowthatyouknowhowtospecifydifferenttypesofinputparameters,let’sseehowyou
cancombinetheminfunctiondefinitions.
Combininginputparameters
Youcancombineinputparameters,aslongasyoufollowtheseorderingrules:
Whendefiningafunction,normalpositionalargumentscomefirst(name),thenany
defaultarguments(name=value),thenthevariablepositionalarguments(*name,or
simply*),thenanykeyword-onlyarguments(eithernameorname=valueformis
WOW! eBook
www.wowebook.org
good),thenanyvariablekeywordarguments(**name).
Ontheotherhand,whencallingafunction,argumentsmustbegiveninthefollowing
order:positionalargumentsfirst(value),thenanycombinationofkeyword
arguments(name=value),variablepositionalarguments(*name),thenvariable
keywordarguments(**name).
Sincethiscanbeabittrickywhenlefthanginginthetheoreticalworld,let’slookata
coupleofquickexamples.
arguments.all.py
deffunc(a,b,c=7,*args,**kwargs):
print('a,b,c:',a,b,c)
print('args:',args)
print('kwargs:',kwargs)
func(1,2,3,*(5,7,9),**{'A':'a','B':'b'})
func(1,2,3,5,7,9,A='a',B='b')#sameaspreviousone
Notetheorderoftheparametersinthefunctiondefinition,andthatthetwocallsare
equivalent.Inthefirstone,we’reusingtheunpackingoperatorsforiterablesand
dictionaries,whileinthesecondonewe’reusingamoreexplicitsyntax.Theexecutionof
thisyields(Iprintedonlytheresultofonecall):
$pythonarguments.all.py
a,b,c:123
args:(5,7,9)
kwargs:{'A':'a','B':'b'}
Let’snowlookatanexamplewithkeyword-onlyarguments.
arguments.all.kwonly.py
deffunc_with_kwonly(a,b=42,*args,c,d=256,**kwargs):
print('a,b:',a,b)
print('c,d:',c,d)
print('args:',args)
print('kwargs:',kwargs)
#bothcallsequivalent
func_with_kwonly(3,42,c=0,d=1,*(7,9,11),e='E',f='F')
func_with_kwonly(3,42,*(7,9,11),c=0,d=1,e='E',f='F')
NotethatIhavehighlightedthekeyword-onlyargumentsinthefunctiondeclaration.They
comeafterthevariablepositionalargument*args,anditwouldbethesameiftheycame
rightafterasingle*(inwhichcasetherewouldn’tbeavariablepositionalargument).The
executionofthisyields(Iprintedonlytheresultofonecall):
$pythonarguments.all.kwonly.py
a,b:342
c,d:01
args:(7,9,11)
kwargs:{'f':'F','e':'E'}
OneotherthingtonotearethenamesIgavetothevariablepositionalandkeyword
WOW! eBook
www.wowebook.org
arguments.You’refreetochoosedifferently,butbeawarethatargsandkwargsarethe
conventionalnamesgiventotheseparameters,atleastgenerically.Nowthatyouknow
howtodefineafunctioninallpossibleflavors,letmeshowyousomethingtricky:
mutabledefaults.
Avoidthetrap!Mutabledefaults
OnethingtobeveryawareofwithPythonisthatdefaultvaluesarecreatedatdeftime,
therefore,subsequentcallstothesamefunctionwillpossiblybehavedifferentlyaccording
tothemutabilityoftheirdefaultvalues.Let’slookatanexample:
arguments.defaults.mutable.py
deffunc(a=[],b={}):
print(a)
print(b)
print('#'*12)
a.append(len(a))#thiswillaffecta'sdefaultvalue
b[len(a)]=len(a)#andthiswillaffectb'sone
func()
func()
func()
Theparametersbothhavemutabledefaultvalues.Thismeansthat,ifyouaffectthose
objects,anymodificationwillstickaroundinsubsequentfunctioncalls.Seeifyoucan
understandtheoutputofthosecalls:
$pythonarguments.defaults.mutable.py
[]
{}
############
[0]
{1:1}
############
[0,1]
{1:1,2:2}
############
It’sinteresting,isn’tit?Whilethisbehaviormayseemveryweirdatfirst,itactuallymakes
sense,andit’sveryhandy,forexample,whenusingmemoizationtechniques(Googlean
exampleofthat,ifyou’reinterested).
Evenmoreinterestingiswhathappenswhen,betweenthecalls,weintroduceonethat
doesn’tusedefaults,likethis:
arguments.defaults.mutable.intermediate.call.py
func()
func(a=[1,2,3],b={'B':1})
func()
Whenwerunthiscode,thisistheoutput:
$pythonarguments.defaults.mutable.intermediate.call.py
[]
WOW! eBook
www.wowebook.org
{}
############
[1,2,3]
{'B':1}
############
[0]
{1:1}
############
Thisoutputshowsusthatthedefaultsareretainedevenifwecallthefunctionwithother
values.Onequestionthatcomestomindis,howdoIgetafreshemptyvalueeverytime?
Well,theconventionisthefollowing:
arguments.defaults.mutable.no.trap.py
deffunc(a=None):
ifaisNone:
a=[]
#dowhateveryouwantwith`a`...
Notethat,byusingtheprecedingtechnique,ifaisn’tpassedwhencallingthefunction,
youalwaysgetabrandnewemptylist.
Okay,enoughwiththeinput,let’slookattheothersideofthecoin,theoutput.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Returnvalues
ReturnvaluesoffunctionsareoneofthosethingswherePythonislightyearsaheadof
mostotherlanguages.Functionsareusuallyallowedtoreturnoneobject(onevalue)but,
inPython,youcanreturnatuple,andthisimpliesthatyoucanreturnwhateveryouwant.
Thisfeatureallowsacodertowritesoftwarethatwouldbemuchhardertowriteinany
otherlanguage,orcertainlymoretedious.We’vealreadysaidthattoreturnsomething
fromafunctionweneedtousethereturnstatement,followedbywhatwewanttoreturn.
Therecanbeasmanyreturnstatementsasneededinthebodyofafunction.
Ontheotherhand,ifwithinthebodyofafunctionwedon’treturnanything,thefunction
willreturnNone.Thisbehaviorisharmlessand,eventhoughIdon’thavetheroomhereto
gointodetailexplainingwhyPythonwasdesignedlikethis,letmejusttellyouthatthis
featureallowsforseveralinterestingpatterns,andconfirmsPythonasaveryconsistent
language.
Isayit’sharmlessbecauseyouareneverforcedtocollecttheresultofafunctioncall.I’ll
showyouwhatImeanwithanexample:
return.none.py
deffunc():
pass
func()#thereturnofthiscallwon'tbecollected.It'slost.
a=func()#thereturnofthisoneinsteadiscollectedinto`a`
print(a)#prints:None
Notethatthewholebodyofthefunctioniscomprisedonlyofthepassstatement.Asthe
officialdocumentationtellsus,passisanulloperation.Whenitisexecuted,nothing
happens.Itisusefulasaplaceholderwhenastatementisrequiredsyntactically,butno
codeneedstobeexecuted.Inotherlanguages,wewouldprobablyjustindicatethatwitha
pairofcurlybraces({}),whichdefineanemptyscopebutinPythonascopeisdefinedby
indentingcode,thereforeastatementsuchaspassisnecessary.
Noticealsothatthefirstcallofthefunctionfuncreturnsavalue(None)whichwedon’t
collect.AsIsaidbefore,collectingthereturnvalueofafunctioncallisnotmandatory.
Now,that’sgoodbutnotveryinterestingso,howaboutwewriteaninterestingfunction?
RememberthatinChapter1,IntroductionandFirstSteps–TakeaDeepBreath,wetalked
aboutthefactorialofafunction.Let’swriteourownhere(forsimplicity,Iwillassumethe
functionisalwayscalledcorrectlywithappropriatevaluessoIwon’tsanity-checkonthe
inputargument):
return.single.value.py
deffactorial(n):
ifnin(0,1):
return1
result=n
forkinrange(2,n):
result*=k
WOW! eBook
www.wowebook.org
returnresult
f5=factorial(5)#f5=120
Notethatwehavetwopointsofreturn.Ifniseither0or1(inPythonit’scommontouse
theintypeofcheckasIdidinsteadofthemoreverboseifn==0orn==1:),we
return1.Otherwise,weperformtherequiredcalculation,andwereturnresult.Canwe
writethisfunctionalittlebitmorePythonically?Yes,butI’llletyoufigureoutthatfor
yourself,asanexercise.
return.single.value.2.py
fromfunctoolsimportreduce
fromoperatorimportmul
deffactorial(n):
returnreduce(mul,range(1,n+1),1)
f5=factorial(5)#f5=120
Iknowwhatyou’rethinking,oneline?Pythoniselegant,andconcise!Ithinkthis
functionisreadableevenifyouhaveneverseenreduceormul,butifyoucan’treaditor
understandit,setasideafewminutesanddosomeresearchonthePythondocumentation
untilitsbehavioriscleartoyou.Beingabletolookupfunctionsinthedocumentationand
understandcodewrittenbysomeoneelseisataskeverydeveloperneedstobeableto
perform,sothinkofthisasagoodexercise,andgoodluck!
Tip
Tothisend,makesureyoulookupthehelpfunction,whichcomesinveryhandy
exploringwiththeconsole.
WOW! eBook
www.wowebook.org
Returningmultiplevalues
Unlikeinmostotherlanguages,inPythonit’sveryeasytoreturnmultipleobjectsfroma
function.Thisfeatureopensupawholeworldofpossibilitiesandallowsyoutocodeina
stylethatishardtoreproducewithotherlanguages.Ourthinkingislimitedbythetools
weuse,thereforewhenPythongivesyoumorefreedomthanotherlanguages,itisactually
boostingyourowncreativityaswell.Toreturnmultiplevaluesisveryeasy,youjustuse
tuples(eitherexplicitlyorimplicitly).Let’slookatasimpleexamplethatmimicsthe
divmodbuilt-infunction:
return.multiple.py
defmoddiv(a,b):
returna//b,a%b
print(moddiv(20,7))#prints(2,6)
Icouldhavewrappedthehighlightedpartintheprecedingcodeinbraces,makingitan
explicittuple,butthere’snoneedforthat.Theprecedingfunctionreturnsboththeresult
andtheremainderofthedivision,atthesametime.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Afewusefultips
Whenwritingfunctions,it’sveryusefultofollowguidelinessothatyouwritethemwell.
I’llquicklypointsomeofthemouthere:
Functionsshoulddoonething:Functionsthatdoonethingareeasytodescribein
oneshortsentence.Functionswhichdomultiplethingscanbesplitintosmaller
functionswhichdoonething.Thesesmallerfunctionsareusuallyeasiertoreadand
understand.Rememberthedatascienceexamplewesawafewpagesago.
Functionsshouldbesmall:Thesmallertheyare,theeasieritistotestthemandto
writethemsothattheydoonething.
Thefewerinputparameters,thebetter:Functionswhichtakealotofarguments
quicklybecomehardertomanage(amongotherissues).
Functionsshouldbeconsistentintheirreturnvalues:ReturningFalseorNoneis
notthesamething,evenifwithinaBooleancontexttheybothevaluatetoFalse.
Falsemeansthatwehaveinformation(False),whileNonemeansthatthereisno
information.Trywritingfunctionswhichreturninaconsistentway,nomatterwhat
happensintheirbody.
Functionsshouldn’thavesideeffects:Inotherwords,functionsshouldnotaffect
thevaluesyoucallthemwith.Thisisprobablythehardeststatementtounderstandat
thispoint,soI’llgiveyouanexampleusinglists.Inthefollowingcode,notehow
numbersisnotsortedbythesortedfunction,whichactuallyreturnsasortedcopyof
numbers.Conversely,thelist.sort()methodisactingonthenumbersobjectitself,
andthatisfinebecauseitisamethod(afunctionthatbelongstoanobjectand
thereforehastherightstomodifyit):
>>>numbers=[4,1,7,5]
>>>sorted(numbers)#won'tsorttheoriginal`numbers`list
[1,4,5,7]
>>>numbers#let'sverify
[4,1,7,5]#good,untouched
>>>numbers.sort()#thiswillactonthelist
>>>numbers
[1,4,5,7]
Followtheseguidelinesandyou’llwritebetterfunctions,whichwillserveyouwell.
Note
Chapter3,FunctionsinCleanCodebyRobertC.Martin,PrenticeHallisdedicatedto
functionsandit’sprobablythebestsetofguidelinesI’veeverreadonthesubject.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Recursivefunctions
Whenafunctioncallsitselftoproducearesult,itissaidtoberecursive.Sometimes
recursivefunctionsareveryusefulinthattheymakeiteasiertowritecode.Some
algorithmsareveryeasytowriteusingtherecursiveparadigm,whileothersarenot.There
isnorecursivefunctionthatcannotberewritteninaniterativefashion,soit’susuallyupto
theprogrammertochoosethebestapproachforthecaseathand.
Arecursivefunctionusuallyhasasetofbasecasesforwhichthereturnvaluedoesn’t
dependonasubsequentcalltothefunctionitselfandasetofrecursivecases,forwhich
thereturnvalueiscalculatedwithoneormorecallstothefunctionitself.
Asanexample,wecanconsiderthe(hopefullyfamiliarbynow)factorialfunctionN!.
ThebasecaseiswhenNiseither0or1.Thefunctionreturns1withnoneedforfurther
calculation.Ontheotherhand,inthegeneralcase,N!returnstheproduct1*2*…*(N1)*N.Ifyouthinkaboutit,N!canberewrittenlikethis:N!=(N-1)!*N.Asapractical
example,consider5!=1*2*3*4*5=(1*2*3*4)*5=4!*5.
Let’swritethisdownincode:
recursive.factorial.py
deffactorial(n):
ifnin(0,1):#basecase
return1
returnfactorial(n-1)*n#recursivecase
Note
Whenwritingrecursivefunctions,alwaysconsiderhowmanynestedcallsyoumake,there
isalimit.Forfurtherinformationonthis,checkoutsys.getrecursionlimit()and
sys.setrecursionlimit().
Recursivefunctionsareusedalotwhenwritingalgorithmsandtheycanbereallyfunto
write.Asagoodexercise,trytosolveacoupleofsimpleproblemsusingbotharecursive
andaniterativeapproach.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Anonymousfunctions
OnelasttypeoffunctionsthatIwanttotalkaboutareanonymousfunctions.These
functions,whicharecalledlambdasinPython,areusuallyusedwhenafully-fledged
functionwithitsownnamewouldbeoverkill,andallwewantisaquick,simpleone-liner
thatdoesthejob.
ImaginethatyouwantalistofallthenumbersuptoNwhicharemultiplesoffive.
Imaginethatyouwanttofilterthoseoutusingthefilterfunction,whichtakesafunction
andaniterableandconstructsafilterobjectwhichyoucaniterateon,fromthoseelements
ofiterableforwhichthefunctionreturnsTrue.Withoutusingananonymousfunction,you
woulddosomethinglikethis:
filter.regular.py
defis_multiple_of_five(n):
returnnotn%5
defget_multiples_of_five(n):
returnlist(filter(is_multiple_of_five,range(n)))
print(get_multiples_of_five(50))
Ihavehighlightedthemainlogicofget_multiples_of_five.Notehowthefilteruses
is_multiple_of_fivetofilterthefirstnnaturalnumbers.Thisseemsabitexcessive,the
taskissimpleandwedon’tneedtokeeptheis_multiple_of_fivefunctionaroundfor
anythingelse.Let’srewriteitusingalambdafunction:
filter.lambda.py
defget_multiples_of_five(n):
returnlist(filter(lambdak:notk%5,range(n)))
print(get_multiples_of_five(50))
Thelogicisexactlythesamebutthefilteringfunctionisnowalambda.Defininga
lambdaisveryeasyandfollowsthisform:func_name=lambda[parameter_list]:
expression.Afunctionobjectisreturned,whichisequivalenttothis:def
func_name([parameter_list]):returnexpression.
Note
Notethatoptionalparametersareindicatedfollowingthecommonsyntaxofwrapping
theminsquarebrackets.
Let’slookatanothercoupleofexamplesofequivalentfunctionsdefinedinthetwoforms:
lambda.explained.py
#example1:adder
defadder(a,b):
returna+b
#isequivalentto:
adder_lambda=lambdaa,b:a+b
#example2:touppercase
defto_upper(s):
WOW! eBook
www.wowebook.org
returns.upper()
#isequivalentto:
to_upper_lambda=lambdas:s.upper()
Theprecedingexamplesareverysimple.Thefirstoneaddstwonumbers,andthesecond
oneproducestheuppercaseversionofastring.NotethatIassignedwhatisreturnedby
thelambdaexpressionstoaname(adder_lambda,to_upper_lambda),butthereisnoneed
forthatwhenyouuselambdasinthewaywedidinthefilterexamplebefore.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Functionattributes
Everyfunctionisafully-fledgedobjectand,assuch,theyhavemanyattributes.Someof
themarespecialandcanbeusedinanintrospectivewaytoinspectthefunctionobjectat
runtime.Thefollowingscriptisanexamplethatshowsallofthemandhowtodisplay
theirvalueforanexamplefunction:
func.attributes.py
defmultiplication(a,b=1):
"""Returnamultipliedbyb."""
returna*b
special_attributes=[
"__doc__","__name__","__qualname__","__module__",
"__defaults__","__code__","__globals__","__dict__",
"__closure__","__annotations__","__kwdefaults__",
]
forattributeinspecial_attributes:
print(attribute,'->',getattr(multiplication,attribute))
Iusedthebuilt-ingetattrfunctiontogetthevalueofthoseattributes.getattr(obj,
attribute)isequivalenttoobj.attributeandcomesinhandywhenweneedtogetan
attributeatruntimeusingitsstringname.Runningthisscriptyields:
$pythonfunc.attributes.py
__doc__->Returnamultipliedbyb.
__name__->multiplication
__qualname__->multiplication
__module__->__main__
__defaults__->(1,)
__code__-><codeobjectmultiplicationat0x7ff529e79300,file
"ch4/func.attributes.py",line1>
__globals__->{...omitted…}
__dict__->{}
__closure__->None
__annotations__->{}
__kwdefaults__->None
Ihaveomittedthevalueofthe__globals__attribute,itwastoobig.Anexplanationof
themeaningofthisattributecanbefoundinthetypessectionofthePythonDataModel
documentationpage.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Built-infunctions
Pythoncomeswithalotofbuilt-infunctions.Theyareavailableanywhereandyoucan
getalistofthembyinspectingthebuiltinmodulewithdir(__builtin__),orbygoing
totheofficialPythondocumentation.Unfortunately,Idon’thavetheroomtogothrough
allofthemhere.Someofthemwe’vealreadyseen,suchasany,bin,bool,divmod,
filter,float,getattr,id,int,len,list,min,print,set,tuple,type,andzip,but
therearemanymore,whichyoushouldreadatleastonce.
Getfamiliarwiththem,experiment,writeasmallpieceofcodeforeachofthem,make
sureyouhavethematthetipofyourfingerssothatyoucanusethemwhenyouneed
them.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Onefinalexample
Beforewefinishoffthischapter,howaboutafinalexample?Iwasthinkingwecould
writeafunctiontogeneratealistofprimenumbersuptoalimit.We’vealreadyseenthe
codeforthissolet’smakeitafunctionand,tokeepitinteresting,let’soptimizeitabit.
Itturnsoutthatyoudon’tneedtodivideitbyallnumbersfrom2toN-1todecideifa
numberNisprime.Youcanstopat
.Moreover,youdon’tneedtotestthedivisionfor
,youcanjustusetheprimesinthatrange.I’llleaveittoyouto
allnumbersfrom2to
figureoutwhythisworks,ifyou’reinterested.Let’sseehowthecodechanges:
primes.py
frommathimportsqrt,ceil
defget_primes(n):
"""Calculatealistofprimesupton(included)."""
primelist=[]
forcandidateinrange(2,n+1):
is_prime=True
root=int(ceil(sqrt(candidate)))#divisionlimit
forprimeinprimelist:#wetryonlytheprimes
ifprime>root:#noneedtocheckanyfurther
break
ifcandidate%prime==0:
is_prime=False
break
ifis_prime:
primelist.append(candidate)
returnprimelist
Thecodeisthesameasinthepreviouschapter.Wehavechangedthedivisionalgorithm
sothatweonlytestdivisibilityusingthepreviouslycalculatedprimesandwestopped
oncethetestingdivisorwasgreaterthantherootofthecandidate.Weusedtheresultlist
primelisttogettheprimesforthedivision.Wecalculatedtherootvalueusingafancy
formula,theintegervalueoftheceilingoftherootofthecandidate.Whileasimpleint(k
**0.5)+1wouldhaveservedourpurposeaswell,theformulaIchoseiscleanerand
requiresmetouseacoupleofimports,whichIwantedtoshowyou.Checkoutthe
functionsinthemathmodule,theyareveryinteresting!
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Documentingyourcode
I’mabigfanofcodethatdoesn’tneeddocumentation.Whenyouprogramcorrectly,
choosetherightnamesandtakecareofthedetails,yourcodeshouldcomeoutasselfexplanatoryanddocumentationshouldnotbeneeded.Sometimesacommentisvery
usefulthough,andsoissomedocumentation.Youcanfindtheguidelinesfor
documentingPythoninPEP257–Docstringconventions,butI’llshowyouthebasics
here.
Pythonisdocumentedwithstrings,whichareaptlycalleddocstrings.Anyobjectcanbe
documented,andyoucanuseeitherone-lineormulti-linedocstrings.One-linersarevery
simple.Theyshouldnotprovideanothersignatureforthefunction,butclearlystateits
purpose.
docstrings.py
defsquare(n):
"""Returnthesquareofanumbern."""
returnn**2
defget_username(userid):
"""Returntheusernameofausergiventheirid."""
returndb.get(user_id=userid).username
Usingtripledouble-quotedstringsallowsyoutoexpandeasilylateron.Usesentencesthat
endinaperiod,anddon’tleaveblanklinesbeforeorafter.
Multi-linecommentsarestructuredinasimilarway.Thereshouldbeaone-linerthat
brieflygivesyouthegistofwhattheobjectisabout,andthenamoreverbosedescription.
Asanexample,Ihavedocumentedafictitiousconnectfunction,usingtheSphinx
notation,inthefollowingexample.
Note
SphinxisprobablythemostwidelyusedtoolforcreatingPythondocumentation.Infact,
theofficialPythondocumentationwaswrittenwithit.It’sdefinitelyworthspendingsome
timecheckingitout.
docstrings.py
defconnect(host,port,user,password):
"""Connecttoadatabase.
ConnecttoaPostgreSQLdatabasedirectly,usingthegiven
parameters.
:paramhost:ThehostIP.
:paramport:Thedesiredport.
:paramuser:Theconnectionusername.
:parampassword:Theconnectionpassword.
:return:Theconnectionobject.
"""
#bodyofthefunctionhere…
WOW! eBook
www.wowebook.org
returnconnection
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Importingobjects
Nowthatyouknowalotaboutfunctions,let’sseehowtousethem.Thewholepointof
writingfunctionsistobeabletolaterreusethem,andthisinPythontranslatesto
importingthemintothenamespaceinwhichyouneedthem.Therearemanydifferent
waystoimportobjectsintoanamespace,butthemostcommononesarejusttwo:import
module_nameandfrommodule_nameimportfunction_name.Ofcourse,thesearequite
simplisticexamples,butbearwithmeforthetimebeing.
Theformimportmodule_namefindsthemodulemodule_nameanddefinesanameforitin
thelocalnamespacewheretheimportstatementisexecuted.
Theformfrommodule_nameimportidentifierisalittlebitmorecomplicatedthan
that,butbasicallydoesthesamething.Itfindsmodule_nameandsearchesforanattribute
(orasubmodule)andstoresareferencetoidentifierinthelocalnamespace.
Bothformshavetheoptiontochangethenameoftheimportedobjectusingtheasclause,
likethis:
frommymoduleimportmyfuncasbetter_named_func
Justtogiveyouaflavorofwhatimportinglookslike,here’sanexamplefromatest
moduleofanumbertheorylibraryIwrotesomeyearsago(it’savailableonBitbucket):
karma/test_nt.py
importunittest#importstheunittestmodule
frommathimportsqrt#importsonefunctionfrommath
fromrandomimportrandint,sample#twoimportsatonce
frommockimportpatch
fromnose.toolsimport(#multilineimport
assert_equal,
assert_list_equal,
assert_not_in,
)
fromkarmaimportnt,utils
IcommentedsomeofthemandIhopeit’seasytofollow.Whenyouhaveastructureof
filesstartingintherootofyourproject,youcanusethedotnotationtogettotheobject
youwanttoimportintoyourcurrentnamespace,beitapackage,amodule,aclass,a
function,oranythingelse.Thefrommoduleimportsyntaxalsoallowsacatch-allclause
frommoduleimport*,whichissometimesusedtogetallthenamesfromamoduleinto
thecurrentnamespaceatonce,butit’sfrowneduponforseveralreasons:performances,
theriskofsilentlyshadowingothernames,andsoon.Youcanreadallthatthereisto
knowaboutimportsintheofficialPythondocumentationbut,beforeweleavethesubject,
letmegiveyouabetterexample.
Imaginethatyouhavedefinedacoupleoffunctions:square(n)andcube(n)inamodule,
funcdef.py,whichisinthelibfolder.Youwanttousetheminacoupleofmodules
WOW! eBook
www.wowebook.org
whichareatthesamelevelofthelibfolder,calledfunc_import.py,andfunc_from.py.
Showingthetreestructureofthatprojectproducessomethinglikethis:
├──func_from.py
├──func_import.py
├──lib
├──funcdef.py
└──__init__.py
BeforeIshowyouthecodeofeachmodule,pleaserememberthatinordertotellPython
thatitisactuallyapackage,weneedtoputa__init__.pymoduleinit.
Note
Therearetwothingstonoteaboutthe__init__.pyfile.Firstofall,itisafullyfledged
Pythonmodulesoyoucanputcodeintoitasyouwouldwithanyothermodule.Second,
asofPython3.3,itspresenceisnolongerrequiredtomakeafolderbeinterpretedasa
Pythonpackage.
Thecodeisasfollows:
funcdef.py
defsquare(n):
returnn**2
defcube(n):
returnn**3
func_import.py
importlib.funcdef
print(lib.funcdef.square(10))
print(lib.funcdef.cube(10))
func_from.py
fromlib.funcdefimportsquare,cube
print(square(10))
print(cube(10))
Boththesefiles,whenexecuted,print100and1000.Youcanseehowdifferentlywethen
accessthesquareandcubefunctions,accordingtohowandwhatweimportedinthe
currentscope.
WOW! eBook
www.wowebook.org
Relativeimports
Theimportswe’veseenuntilnowarecalledabsolute,thatistosaytheydefinethewhole
pathofthemodulethatwewanttoimport,orfromwhichwewanttoimportanobject.
ThereisanotherwayofimportingobjectsintoPython,whichiscalledrelativeimport.It’s
helpfulinsituationsinwhichwewanttorearrangethestructureoflargepackageswithout
havingtoeditsub-packages,orwhenwewanttomakeamoduleinsideapackageableto
importitself.Relativeimportsaredonebyaddingasmanyleadingdotsinfrontofthe
moduleasthenumberoffoldersweneedtobacktrack,inordertofindwhatwe’re
searchingfor.Simplyput,itissomethinglikethis:
from.mymoduleimportmyfunc
Foracompleteexplanationofrelativeimports,refertoPEP328
(https://www.python.org/dev/peps/pep-0328).
Inlaterchapters,we’llcreateprojectsusingdifferentlibrariesandwe’lluseseveral
differenttypesofimports,includingrelativeones,somakesureyoutakeabitoftimeto
readupaboutitintheofficialPythondocumentation.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,finallyweexploredtheworldoffunctions.Theyareextremelyimportant
and,fromnowon,we’llusethembasicallyeverywhere.Wetalkedaboutthemainreasons
forusingthem,themostimportantofwhicharecodereuseandimplementationhiding.
Wesawthatafunctionobjectislikeaboxthattakesoptionalinputandproducesoutput.
Wecanfeedinputvaluestoafunctioninmanydifferentways,usingpositionaland
keywordarguments,andusingvariablesyntaxforbothtypes.
Nowyoushouldknowhowtowriteafunction,howtodocumentit,importitintoyour
code,andcallit.
ThenextchapterwillforcemetopushmyfootdownonthethrottleevenmoresoI
suggestyoutakeanyopportunityyougettoconsolidateandenrichtheknowledgeyou’ve
gathereduntilnowbyputtingyournoseintothePythonofficialdocumentation.
Readyforthecoolstuff?Let’sgo!
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter5.SavingTimeandMemory
“It’snotthedailyincreasebutdailydecrease.Hackawayattheunessential.” —BruceLee
IlovethisquotefromBruceLee,hewassuchawiseman!Especially,thesecondpart,
hackawayattheunessential,istomewhatmakesacomputerprogramelegant.Afterall,
ifthereisabetterwayofdoingthingssothatwedon’twastetimeormemory,whynot?
Sometimes,therearevalidreasonsfornotpushingourcodeuptothemaximumlimit:for
example,sometimestoachieveanegligibleimprovement,wehavetosacrificeon
readabilityormaintainability.Doesitmakeanysensetohaveawebpageservedin1
secondwithunreadable,complicatedcode,whenwecanserveitin1.05secondswith
readable,cleancode?No,itmakesnosense.
Ontheotherhand,sometimesit’sperfectlylicittotryandshaveoffamillisecondfroma
function,especiallywhenthefunctionismeanttobecalledthousandsoftimes.Every
millisecondyousavetheremeansonesecondsavedperthousandofcalls,andthiscould
bemeaningfulforyourapplication.
Inlightoftheseconsiderations,thefocusofthischapterwillnotbetogiveyouthetools
topushyourcodetotheabsolutelimitsofperformanceandoptimization“nomatter
what”,butrather,togiveyouthetoolstowriteefficient,elegantcodethatreadswell,runs
fast,anddoesn’twasteresourcesinanobviousway.
Inthischapter,Iwillperformseveralmeasurementsandcomparisons,andcautiously
drawsomeconclusions.Pleasedokeepinmindthatonadifferentboxwithadifferent
setuporadifferentoperatingsystem,resultsmayvary.Takealookatthiscode:
squares.py
defsquare1(n):
returnn**2#squaringthroughthepoweroperator
defsquare2(n):
returnn*n#squaringthroughmultiplication
Bothfunctionsreturnthesquareofn,butwhichisfaster?FromasimplebenchmarkIran
onthem,itlookslikethesecondisslightlyfaster.Ifyouthinkaboutit,itmakessense:
calculatingthepowerofanumberinvolvesmultiplicationandtherefore,whatever
algorithmyoumayusetoperformthepoweroperation,it’snotlikelytobeatasimple
multiplicationliketheoneinsquare2.
Dowecareaboutthisresult?Inmostcasesno.Ifyou’recodingane-commercewebsite,
chancesareyouwon’teverevenneedtoraiseanumbertothesecondpower,andifyou
do,youprobablywillhavetodoitafewtimesperpage.Youdon’tneedtoconcern
yourselfonsavingafewmicrosecondsonafunctionyoucallafewtimes.
So,whendoesoptimizationbecomeimportant?Oneverycommoncaseiswhenyouhave
todealwithhugecollectionsofdata.Ifyou’reapplyingthesamefunctiononamillion
WOW! eBook
www.wowebook.org
customerobjects,thenyouwantyourfunctiontobetuneduptoitsbest.Gaining1/10ofa
secondonafunctioncalledonemilliontimessavesyou100,000seconds,whichareabout
27.7hours.That’snotthesame,right?So,let’sfocusoncollections,andlet’sseewhich
toolsPythongivesyoutohandlethemwithefficiencyandgrace.
Note
Manyoftheconceptswewillseeinthischapterarebasedonthoseofiteratorand
iterable.Simplyput,theabilityforanobjecttoreturnitsnextelementwhenasked,andto
raiseaStopIterationexceptionwhenexhausted.We’llseehowtocodeacustomiterator
anditerableobjectsinthenextchapter.
WOW! eBook
www.wowebook.org
map,zip,andfilter
We’llstartbyreviewingmap,filter,andzip,whicharethemainbuilt-infunctionsone
canemploywhenhandlingcollections,andthenwe’lllearnhowtoachievethesame
resultsusingtwoveryimportantconstructs:comprehensionsandgenerators.Fastenyour
seatbelt!
WOW! eBook
www.wowebook.org
map
AccordingtotheofficialPythondocumentation:
map(function,iterable,...)returnsaniteratorthatappliesfunctiontoevery
itemofiterable,yieldingtheresults.Ifadditionaliterableargumentsarepassed,
functionmusttakethatmanyargumentsandisappliedtotheitemsfromalliterables
inparallel.Withmultipleiterables,theiteratorstopswhentheshortestiterableis
exhausted.
Wewillexplaintheconceptofyieldinglateroninthechapter.Fornow,let’stranslatethis
intocode:we’llusealambdafunctionthattakesavariablenumberofpositional
arguments,andjustreturnsthemasatuple.Also,asmapreturnsaniterator,we’llneedto
wrapeachcalltoitwithinalistconstructorsothatweexhausttheiterablebyputtingall
ofitselementsintoalist(you’llseeanexampleofthisinthecode):
map.example.py
>>>map(lambda*a:a,range(3))#withoutwrappinginlist…
<mapobjectat0x7f563513b518>#wegettheiteratorobject
>>>list(map(lambda*a:a,range(3)))#wrappinginlist…
[(0,),(1,),(2,)]#wegetalistwithitselements
>>>list(map(lambda*a:a,range(3),'abc'))#2iterables
[(0,'a'),(1,'b'),(2,'c')]
>>>list(map(lambda*a:a,range(3),'abc',range(4,7)))#3
[(0,'a',4),(1,'b',5),(2,'c',6)]
>>>#mapstopsattheshortestiterator
>>>list(map(lambda*a:a,(),'abc'))#emptytupleisshortest
[]
>>>list(map(lambda*a:a,(1,2),'abc'))#(1,2)shortest
[(1,'a'),(2,'b')]
>>>list(map(lambda*a:a,(1,2,3,4),'abc'))#'abc'shortest
[(1,'a'),(2,'b'),(3,'c')]
Intheprecedingcodeyoucanseewhy,inordertopresentyouwiththeresults,Ihaveto
wrapthecallstomapwithinalistconstructor,otherwiseIgetthestringrepresentationof
amapobject,whichisnotreallyusefulinthiscontext,isit?
Youcanalsonoticehowtheelementsofeachiterableareappliedtothefunction:atfirst,
thefirstelementofeachiterable,thenthesecondoneofeachiterable,andsoon.Notice
alsothatmapstopswhentheshortestoftheiterableswecalleditwithisexhausted.Thisis
actuallyaverynicebehavior:itdoesn’tforceustoleveloffalltheiterablestoacommon
length,anditdoesn’tbreakiftheyaren’tallthesamelength.
mapisveryusefulwhenyouhavetoapplythesamefunctiontooneormorecollectionsof
objects.Asamoreinterestingexample,let’sseethedecorate-sort-undecorateidiom
(alsoknownasSchwartziantransform).It’satechniquethatwasextremelypopular
whenPythonsortingwasn’tprovidingkey-functions,andthereforetodayislessused,but
it’sacooltrickthatstillcomesathandonceinawhile.
Let’sseeavariationofitinthenextexample:wewanttosortindescendingorderbythe
WOW! eBook
www.wowebook.org
sumofcreditsaccumulatedbystudents,sotohavethebeststudentatposition0.Wewrite
afunctiontoproduceadecoratedobject,wesort,andthenweundecorate.Eachstudent
hascreditsinthree(possiblydifferent)subjects.Todecorateanobjectmeanstotransform
it,eitheraddingextradatatoit,orputtingitintoanotherobject,inawaythatallowsusto
beabletosorttheoriginalobjectsthewaywewant.Afterthesorting,werevertthe
decoratedobjectstogettheoriginalonesfromthem.Thisiscalledtoundecorate.
decorate.sort.undecorate.py
students=[
dict(id=0,credits=dict(math=9,physics=6,history=7)),
dict(id=1,credits=dict(math=6,physics=7,latin=10)),
dict(id=2,credits=dict(history=8,physics=9,chemistry=10)),
dict(id=3,credits=dict(math=5,physics=5,geography=7)),
]
defdecorate(student):
#createa2-tuple(sumofcredits,student)fromstudentdict
return(sum(student['credits'].values()),student)
defundecorate(decorated_student):
#discardsumofcredits,returnoriginalstudentdict
returndecorated_student[1]
students=sorted(map(decorate,students),reverse=True)
students=list(map(undecorate,students))
Intheprecedingcode,Ihighlightedthetrickyandimportantparts.Let’sstartby
understandingwhateachstudentobjectis.Infact,let’sprintthefirstone:
{'credits':{'history':7,'math':9,'physics':6},'id':0}
Youcanseethatit’sadictionarywithtwokeys:idandcredit.Thevalueofcreditis
alsoadictionaryinwhichtherearethreesubject/gradekey/valuepairs.AsI’msureyou
recallfromourvisitinthedatastructuresworld,callingdict.values()returnsanobject
similartoaniterable,withonlythevalues.Therefore,
sum(student['credits'].values()),forthefirststudentisequivalenttosum(9,6,7)
(oranypermutationofthosenumbersbecausedictionariesdon’tretainorder,butluckily
forus,additioniscommutative).
Withthatoutoftheway,it’seasytoseewhatistheresultofcallingdecoratewithanyof
thestudents.Let’sprinttheresultofdecorate(students[0]):
(22,{'credits':{'history':7,'math':9,'physics':6},'id':0})
That’snice!Ifwedecorateallthestudentslikethis,wecansortthemontheirtotalamount
ofcreditsbutjustsortingthelistoftuples.Inordertoapplythedecorationtoeachitemin
students,wecallmap(decorate,students).Thenwesorttheresult,andthenwe
undecorateinasimilarfashion.Ifyouhavegonethroughthepreviouschapterscorrectly,
understandingthiscodeshouldn’tbetoohard.
Printingstudentsafterrunningthewholecodeyields:
$pythondecorate.sort.undecorate.py
WOW! eBook
www.wowebook.org
[{'credits':{'chemistry':10,'history':8,'physics':9},'id':2},
{'credits':{'latin':10,'math':6,'physics':7},'id':1},
{'credits':{'history':7,'math':9,'physics':6},'id':0},
{'credits':{'geography':7,'math':5,'physics':5},'id':3}]
Andyoucansee,bytheorderofthestudentobjects,thattheyhaveindeedbeensortedby
thesumoftheircredits.
Note
Formoreonthedecorate-sort-undecorateidiom,there’saveryniceintroductioninthe
sortinghow-tosectionoftheofficialPythondocumentation
(https://docs.python.org/3.4/howto/sorting.html#the-old-way-using-decorate-sortundecorate).
Onethingtonoticeaboutthesortingpart:whatiftwoormorestudentssharethesame
totalsum?Thesortingalgorithmwouldthenproceedsortingthetuplesbycomparingthe
studentobjectswitheachother.Thisdoesn’tmakeanysense,andinmorecomplexcases
couldleadtounpredictableresults,orevenerrors.Ifyouwanttobesuretoavoidthis
issue,onesimplesolutionistocreatea3-tupleinsteadofa2-tuple,havingthesumof
creditsinthefirstposition,thepositionofthestudentobjectinthestudentslistinthe
secondone,andthestudentobjectitselfinthethirdone.Thisway,ifthesumofcreditsis
thesame,thetupleswillbesortedagainsttheposition,whichwillalwaysbedifferentand
thereforeenoughtoresolvethesortingbetweenanypairoftuples.Formore
considerationsonthistopic,pleasecheckoutthesortinghow-tosectionontheofficial
Pythondocumentation.
WOW! eBook
www.wowebook.org
zip
We’vealreadycoveredzipinthepreviouschapters,solet’sjustdefineitproperlyand
thenIwanttoshowyouhowyoucouldcombineitwithmap.
AccordingtothePythondocumentation:
zip(*iterables)returnsaniteratoroftuples,wherethei-thtuplecontainsthei-th
elementfromeachoftheargumentsequencesoriterables.Theiteratorstopswhen
theshortestinputiterableisexhausted.Withasingleiterableargument,itreturnsan
iteratorof1-tuples.Withnoarguments,itreturnsanemptyiterator.
Let’sseeanexample:
zip.grades.py
>>>grades=[18,23,30,27,15,9,22]
>>>avgs=[22,21,29,24,18,18,24]
>>>list(zip(avgs,grades))
[(22,18),(21,23),(29,30),(24,27),(18,15),(18,9),(24,22)]
>>>list(map(lambda*a:a,avgs,grades))#equivalenttozip
[(22,18),(21,23),(29,30),(24,27),(18,15),(18,9),(24,22)]
Intheprecedingcode,we’rezippingtogethertheaverageandthegradeforthelastexam,
pereachstudent.Noticehowthecodeinsidethetwolistcallsproducesexactlythesame
result,showinghoweasyitistoreproducezipusingmap.Noticealsothat,aswedofor
map,wehavetofeedtheresultofthezipcalltoalistconstructor.
Asimpleexampleonthecombineduseofmapandzipcouldbeawayofcalculatingthe
element-wisemaximumamongstsequences,thatis,themaximumofthefirstelementof
eachsequence,thenthemaximumofthesecondone,andsoon:
maxims.py
>>>a=[5,9,2,4,7]
>>>b=[3,7,1,9,2]
>>>c=[6,8,0,5,3]
>>>maxs=map(lambdan:max(*n),zip(a,b,c))
>>>list(maxs)
[6,9,2,9,7]
Noticehoweasyitistocalculatethemaxvaluesofthreesequences.zipisnotstrictly
neededofcourse,wecouldjustusemap,butthiswouldrequireustowriteamuchmore
complicatedfunctiontofeedmapwith.Sometimeswemaybeinasituationwhere
changingthefunctionwefeedtomapisnotevenpossible.Incaseslikethese,beingableto
massagethedata(likewe’redoinginthisexamplewithzip)isveryhelpful.
WOW! eBook
www.wowebook.org
filter
AccordingtothePythondocumentation:
filter(function,iterable)constructaniteratorfromthoseelementsofiterable
forwhichfunctionreturnsTrue.iterablemaybeeitherasequence,acontainerwhich
supportsiteration,oraniterator.IffunctionisNone,theidentityfunctionisassumed,
thatis,allelementsofiterablethatarefalseareremoved.
Let’sseeaveryquickexample:
filter.py
>>>test=[2,5,8,0,0,1,0]
>>>list(filter(None,test))
[2,5,8,1]
>>>list(filter(lambdax:x,test))#equivalenttopreviousone
[2,5,8,1]
>>>list(filter(lambdax:x>4,test))#keeponlyitems>4
[5,8]
Intheprecedingcode,noticehowthesecondcalltofilterisequivalenttothefirstone.If
wepassafunctionthattakesoneargumentandreturnstheargumentitself,onlythose
argumentsthatareTruewillmakethefunctionreturnTrue,thereforethisbehavioris
exactlythesameaspassingNone.It’softenaverygoodexercisetomimicsomeofthe
built-inPythonbehaviors.Whenyousucceedyoucansayyoufullyunderstandhow
Pythonbehavesinaspecificsituation.
Armedwithmap,zip,andfilter(andseveralotherfunctionsfromthePythonstandard
library)wecanmassagesequencesveryeffectively.Butthosefunctionsarenottheonly
waytodoit.Solet’sseeoneofthenicestfeaturesofPython:comprehensions.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Comprehensions
Pythonoffersyoudifferenttypesofcomprehensions:list,dict,andset.
We’llconcentrateonthefirstonefornow,andthenitwillbeeasytoexplaintheother
two.
Alistcomprehensionisaquickwayofmakingalist.Usuallythelististheresultof
someoperationthatmayinvolveapplyingafunction,filtering,orbuildingadifferentdata
structure.
Let’sstartwithaverysimpleexampleIwanttocalculatealistwiththesquaresofthefirst
10naturalnumbers.Howwouldyoudoit?Thereareacoupleofequivalentways:
squares.map.py
#IfyoucodelikethisyouarenotaPythonguy!;)
>>>squares=[]
>>>forninrange(10):
...squares.append(n**2)
...
>>>list(squares)
[0,1,4,9,16,25,36,49,64,81]
#Thisisbetter,oneline,niceandreadable
>>>squares=map(lambdan:n**2,range(10))
>>>list(squares)
[0,1,4,9,16,25,36,49,64,81]
Theprecedingexampleshouldbenothingnewforyou.Let’sseehowtoachievethesame
resultusingalistcomprehension:
squares.comprehension.py
>>>[n**2forninrange(10)]
[0,1,4,9,16,25,36,49,64,81]
Assimpleasthat.Isn’titelegant?Basicallywehaveputaforloopwithinsquare
brackets.Let’snowfilterouttheoddsquares.I’llshowyouhowtodoitwithmapand
filter,andthenusingalistcomprehensionagain.
even.squares.py
#usingmapandfilter
sq1=list(
filter(lambdan:notn%2,map(lambdan:n**2,range(10)))
)
#equivalent,butusinglistcomprehensions
sq2=[n**2forninrange(10)ifnotn%2]
print(sq1,sq1==sq2)#prints:[0,4,16,36,64]True
Ithinkthatnowthedifferenceinreadabilityisevident.Thelistcomprehensionreads
muchbetter.It’salmostEnglish:givemeallsquares(n**2)fornbetween0and9ifnis
even.
WOW! eBook
www.wowebook.org
AccordingtothePythondocumentation:
Alistcomprehensionconsistsofbracketscontaininganexpressionfollowedbyafor
clause,thenzeroormorefororifclauses.Theresultwillbeanewlistresulting
fromevaluatingtheexpressioninthecontextoftheforandifclauseswhichfollow
it”.
WOW! eBook
www.wowebook.org
Nestedcomprehensions
Let’sseeanexampleofnestedloops.It’sverycommonwhendealingwithalgorithmsto
havetoiterateonasequenceusingtwoplaceholders.Thefirstonerunsthroughthewhole
sequence,lefttoright.Thesecondoneaswell,butitstartsfromthefirstone,insteadof0.
Theconceptisthatoftestingallpairswithoutduplication.Let’sseetheclassicalforloop
equivalent.
pairs.for.loop.py
items='ABCDE'
pairs=[]
forainrange(len(items)):
forbinrange(a,len(items)):
pairs.append((items[a],items[b]))
Ifyouprintpairsattheend,youget:
[('A','A'),('A','B'),('A','C'),('A','D'),('A','E'),('B','B'),
('B','C'),('B','D'),('B','E'),('C','C'),('C','D'),('C','E'),
('D','D'),('D','E'),('E','E')]
Allthetupleswiththesameletterarethoseforwhichbisatthesamepositionasa.Now,
let’sseehowwecantranslatethisinalistcomprehension:
pairs.list.comprehension.py
items='ABCDE'
pairs=[(items[a],items[b])
forainrange(len(items))forbinrange(a,len(items))]
Thisversionisjusttwolineslongandachievesthesameresult.Noticethatinthis
particularcase,becausetheforloopoverbhasadependencyona,itmustfollowthefor
loopoverainthecomprehension.Ifyouswapthemaround,you’llgetanameerror.
WOW! eBook
www.wowebook.org
Filteringacomprehension
Wecanapplyfilteringtoacomprehension.Let’sfirstdoitwithfilter.Let’sfindall
Pythagoreantripleswhoseshortsidesarenumberssmallerthan10.Weobviouslydon’t
wanttotestacombinationtwice,andthereforewe’lluseatrickliketheonewesawinthe
previousexample.
Note
APythagoreantripleisatriple(a,b,c)ofintegernumberssatisfyingtheequation
.
pythagorean.triple.py
frommathimportsqrt
#thiswillgenerateallpossiblepairs
mx=10
legs=[(a,b,sqrt(a**2+b**2))
forainrange(1,mx)forbinrange(a,mx)]
#thiswillfilteroutallnonpythagoreantriples
legs=list(
filter(lambdatriple:triple[2].is_integer(),legs))
print(legs)#prints:[(3,4,5.0),(6,8,10.0)]
Intheprecedingcode,wegeneratedalistof3-tuples,legs.Eachtuplecontainstwo
integernumbers(thelegs)andthehypotenuseofthePythagoreantrianglewhoselegsare
thefirsttwonumbersinthetuple.Forexample,whena=3andb=4,thetuplewillbe(3,
4,5.0),andwhena=5andb=7,thetuplewillbe(5,7,8.602325267042627).
Afterhavingallthetriplesdone,weneedtofilteroutallthosethatdon’thavea
hypotenusethatisanintegernumber.Inordertodothis,wefilterbasedon
float_number.is_integer()beingTrue.ThismeansthatofthetwoexampletuplesI
showedyoubefore,theonewithhypotenuse5.0willberetained,whiletheonewith
hypotenuse8.602325267042627willbediscarded.
Thisisgood,butIdon’tlikethatthetriplehastwointegernumbersandafloat.Theyare
supposedtobeallintegers,solet’susemaptofixthis:
pythagorean.triple.int.py
frommathimportsqrt
mx=10
legs=[(a,b,sqrt(a**2+b**2))
forainrange(1,mx)forbinrange(a,mx)]
legs=filter(lambdatriple:triple[2].is_integer(),legs)
#thiswillmakethethirdnumberinthetuplesinteger
legs=list(
map(lambdatriple:triple[:2]+(int(triple[2]),),legs))
print(legs)#prints:[(3,4,5),(6,8,10)]
Noticethestepweadded.Wetakeeachelementinlegsandwesliceit,takingonlythe
firsttwoelementsinit.Then,weconcatenatetheslicewitha1-tuple,inwhichweputthe
integerversionofthatfloatnumberthatwedidn’tlike.
WOW! eBook
www.wowebook.org
Seemslikealotofwork,right?Indeeditis.Let’sseehowtodoallthiswithalist
comprehension:
pythagorean.triple.comprehension.py
frommathimportsqrt
#thisstepisthesameasbefore
mx=10
legs=[(a,b,sqrt(a**2+b**2))
forainrange(1,mx)forbinrange(a,mx)]
#herewecombinefilterandmapinoneCLEANlistcomprehension
legs=[(a,b,int(c))fora,b,cinlegsifc.is_integer()]
print(legs)#prints:[(3,4,5),(6,8,10)]
Iknow.It’smuchbetter,isn’tit?It’sclean,readable,shorter.Inotherwords,elegant.
Tip
I’mgoingquitefasthere,asanticipatedinthesummaryofthelastchapter.Areyou
playingwiththiscode?Ifnot,Isuggestyoudo.It’sveryimportantthatyouplayaround,
breakthings,changethings,seewhathappens.Makesureyouhaveaclearunderstanding
ofwhatisgoingon.Youwanttobecomeaninja,right?
WOW! eBook
www.wowebook.org
dictcomprehensions
Dictionaryandsetcomprehensionsworkexactlylikethelistones,onlythereisalittle
differenceinthesyntax.Thefollowingexamplewillsufficetoexplaineverythingyou
needtoknow:
dictionary.comprehensions.py
fromstringimportascii_lowercase
lettermap=dict((c,k)fork,cinenumerate(ascii_lowercase,1))
Ifyouprintlettermap,youwillseethefollowing(Iomittedthemiddleresults,youget
thegist):
{'a':1,
'b':2,
'c':3,
...omittedresults…
'x':24,
'y':25,
'z':26}
Whathappensintheprecedingcodeisthatwe’refeedingthedictconstructorwitha
comprehension(technically,ageneratorexpression,we’llseeitinabit).Wetellthedict
constructortomakekey/valuepairsfromeachtupleinthecomprehension.Weenumerate
thesequenceofalllowercaseASCIIletters,startingfrom1,usingenumerate.Pieceof
cake.Thereisalsoanotherwaytodothesamething,whichisclosertotheother
dictionarysyntax:
lettermap={c:kfork,cinenumerate(ascii_lowercase,1)}
Itdoesexactlythesamething,withaslightlydifferentsyntaxthathighlightsabitmoreof
thekey:valuepart.
Dictionariesdonotallowduplicationinthekeys,asshowninthefollowingexample:
dictionary.comprehensions.duplicates.py
word='Hello'
swaps={c:c.swapcase()forcinword}
print(swaps)#prints:{'o':'O','l':'L','e':'E','H':'h'}
Wecreateadictionarywithkeys,thelettersinthestring'Hello',andvaluesofthesame
letters,butwiththecaseswapped.Noticethereisonlyone'l':'L'pair.Theconstructor
doesn’tcomplain,simplyreassignsduplicatestothelatestvalue.Let’smakethisclearer
withanotherexample;let’sassigntoeachkeyitspositioninthestring:
dictionary.comprehensions.positions.py
word='Hello'
positions={c:kfork,cinenumerate(word)}
print(positions)#prints:{'l':3,'o':4,'e':1,'H':0}
Noticethevalueassociatedtotheletter'l':3.Thepair'l':2isn’tthere,ithasbeen
overriddenby'l':3.
WOW! eBook
www.wowebook.org
setcomprehensions
Setcomprehensionsareverysimilartolistanddictionaryones.Pythonallowsboththe
set()constructortobeused,ortheexplicit{}syntax.Let’sseeonequickexample:
set.comprehensions.py
word='Hello'
letters1=set(cforcinword)
letters2={cforcinword}
print(letters1)#prints:{'l','o','H','e'}
print(letters1==letters2)#prints:True
Noticehowforsetcomprehensions,asfordictionaries,duplicationisnotallowedand
thereforetheresultingsethasonlyfourletters.Also,noticethattheexpressionsassigned
toletters1andletters2produceequivalentsets.
Thesyntaxusedtocreateletters2isverysimilartotheonewecanusetocreatea
dictionarycomprehension.Youcanspotthedifferenceonlybythefactthatdictionaries
requirekeysandvalues,separatedbycolumns,whilesetsdon’t.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Generators
GeneratorsareoneverypowerfultoolthatPythongiftsuswith.Theyarebasedonthe
conceptsofiteration,aswesaidbefore,andtheyallowforcodingpatternsthatcombine
elegancewithefficiency.
Generatorsareoftwotypes:
Generatorfunctions:Theseareverysimilartoregularfunctions,butinsteadof
returningresultsthroughreturnstatements,theyuseyield,whichallowsthemto
suspendandresumetheirstatebetweeneachcall
Generatorexpressions:Theseareverysimilartothelistcomprehensionswe’ve
seeninthischapter,butinsteadofreturningalisttheyreturnanobjectthatproduces
resultsonebyone
WOW! eBook
www.wowebook.org
Generatorfunctions
Generatorfunctionscomeunderallaspectslikeregularfunctions,withonedifference:
insteadofcollectingresultsandreturningthematonce,theycanstartthecomputation,
yieldonevalue,suspendtheirstatesavingeverythingtheyneedtobeabletoresumeand,
ifcalledagain,resumeandperformanotherstep.Generatorfunctionsareautomatically
turnedintotheirowniteratorsbyPython,soyoucancallnextonthem.
Thisisallverytheoreticalso,let’smakeitclearwhysuchamechanismissopowerful,
andthenlet’sseeanexample.
SayIaskedyoutocountoutloudfrom1toamillion.Youstart,andatsomepointIask
youtostop.Aftersometime,Iaskyoutoresume.Atthispoint,whatistheminimum
informationyouneedtobeabletoresumecorrectly?Well,youneedtorememberthelast
numberyoucalled.IfIstoppedyouafter31415,youwilljustgoonwith31416,andso
on.
Thepointis,youdon’tneedtorememberallthenumbersyousaidbefore31415,nordo
youneedthemtobewrittendownsomewhere.Well,youmaynotknowit,butyou’re
behavinglikeageneratoralready!
Takeagoodlookatthefollowingcode:
first.n.squares.py
defget_squares(n):#classicfunctionapproach
return[x**2forxinrange(n)]
print(get_squares(10))
defget_squares_gen(n):#generatorapproach
forxinrange(n):
yieldx**2#weyield,wedon'treturn
print(list(get_squares_gen(10)))
Theresultoftheprintswillbethesame:[0,1,4,9,16,25,36,49,64,81].But
thereisahugedifferencebetweenthetwofunctions.get_squaresisaclassicfunction
thatcollectsallthesquaresofnumbersin[0,n)inalist,andreturnsit.Ontheotherhand,
get_squares_genisagenerator,andbehavesverydifferently.Eachtimetheinterpreter
reachestheyieldline,itsexecutionissuspended.Theonlyreasonthoseprintsreturnthe
sameresultisbecausewefedget_squares_gentothelistconstructor,whichwhen
calledlikethatexhauststhegeneratorcompletelybyaskingthenextelementuntila
StopIterationisraised.Let’sseethisindetail:
first.n.squares.manual.py
defget_squares_gen(n):
forxinrange(n):
yieldx**2
squares=get_squares_gen(4)#thiscreatesageneratorobject
print(squares)#<generatorobjectget_squares_genat0x7f158…>
print(next(squares))#prints:0
print(next(squares))#prints:1
WOW! eBook
www.wowebook.org
print(next(squares))#prints:4
print(next(squares))#prints:9
#thefollowingraisesStopIteration,thegeneratorisexhausted,
#anyfurthercalltonextwillkeepraisingStopIteration
print(next(squares))
Intheprecedingcode,eachtimewecallnextonthegeneratorobject,weeitherstartit
(firstnext)ormakeitresumefromthelastsuspensionpoint(anyothernext).
Thefirsttimewecallnextonit,weget0,whichisthesquareof0,then1,then4,then9
andsincetheforloopstopsafterthat(nis4),thenthegeneratornaturallyends.Aclassic
functionwouldatthatpointjustreturnNone,butinordertocomplywiththeiteration
protocol,ageneratorwillinsteadraiseaStopIterationexception.
Thisexplainshowaforloopworksforexample.Whenyoucallforkinrange(n),
whathappensunderthehoodisthattheforloopgetsaniteratoroutofrange(n)and
startscallingnextonit,untilStopIterationisraised,whichtellstheforloopthatthe
iterationhasreacheditsend.
Havingthisbehaviorbuilt-inineveryiterationaspectofPythonmakesgeneratorseven
morepowerfulbecauseoncewewritethem,we’llbeabletoplugtheminwhatever
iterationmechanismwewant.
Atthispoint,you’reprobablyaskingyourselfwhywouldyouwanttouseagenerator
insteadofaregularfunction.Well,thetitleofthischaptershouldsuggesttheanswer.I’ll
talkaboutperformanceslater,sofornowlet’sconcentrateonanotheraspect:sometimes
generatorsallowyoutodosomethingthatwouldn’tbepossiblewithasimplelist.For
example,sayyouwanttoanalyzeallpermutationsofasequence.Ifthesequencehas
lengthN,thenthenumberofitspermutationsisN!.Thismeansthatifthesequenceis10
elementslong,thenumberofpermutationsis3628800.Butasequenceof20elements
wouldhave2432902008176640000permutations.Theygrowfactorially.
Nowimagineyouhaveaclassicfunctionthatisattemptingtocalculateallpermutations,
puttheminalist,andreturnittoyou.With10elements,itwouldrequireprobablyafew
tensofseconds,butfor20elementsthereissimplynowaythatitcanbedone.
Ontheotherhand,ageneratorfunctionwillbeabletostartthecomputationandgiveyou
backthefirstpermutation,thenthesecond,andsoon.Ofcourseyouwon’thavethetime
toparsethemall,theyaretoomany,butatleastyou’llbeabletoworkwithsomeofthem.
Rememberwhenweweretalkingaboutthebreakstatementinforloops?Whenwefound
anumberdividingacandidateprimewewerebreakingtheloop,noneedtogoon.
Sometimesit’sexactlythesame,onlytheamountofdatayouhavetoiterateoverisso
hugethatyoucannotkeepitallinmemoryinalist.Inthiscase,generatorsareinvaluable:
theymakepossiblewhatwouldn’tbepossibleotherwise.
So,inordertosavememory(andtime),usegeneratorfunctionswheneverpossible.
It’salsoworthnotingthatyoucanusethereturnstatementinageneratorfunction.Itwill
produceaStopIterationexceptiontoberaised,effectivelyendingtheiteration.Thisis
extremelyimportant.Ifareturnstatementwereactuallytomakethefunctionreturn
WOW! eBook
www.wowebook.org
something,itwouldbreaktheiterationprotocol.Pythonconsistencypreventsthis,and
allowsusgreateasewhencoding.Let’sseeaquickexample:
gen.yield.return.py
defgeometric_progression(a,q):
k=0
whileTrue:
result=a*q**k
ifresult<=100000:
yieldresult
else:
return
k+=1
forningeometric_progression(2,5):
print(n)
,
,….
Theprecedingcodeyieldsalltermsofthegeometricprogressiona,aq,
Whentheprogressionproducesatermthatisgreaterthan100,000,thegeneratorstops
(withareturnstatement).Runningthecodeproducesthefollowingresult:
$pythongen.yield.return.py
2
10
50
250
1250
6250
31250
Thenexttermwouldhavebeen156250,whichistoobig.
WOW! eBook
www.wowebook.org
Goingbeyondnext
Atthebeginningofthischapter,Itoldyouthatgeneratorobjectsarebasedontheiteration
protocol.We’llseeinthenextchapteracompleteexampleofhowtowriteacustom
iterator/iterableobject.Fornow,Ijustwantyoutounderstandhownext()works.
Whathappenswhenyoucallnext(generator)isthatyou’recallingthe
generator.__next__()method.Remember,amethodisjustafunctionthatbelongstoan
object,andobjectsinPythoncanhavespecialmethods.Ourfriend__next__()isjustone
oftheseanditspurposeistoreturnthenextelementoftheiteration,ortoraise
StopIterationwhentheiterationisoverandtherearenomoreelementstoreturn.
Note
InPython,anobject’sspecialmethodsarealsocalledmagicmethods,ordunder(from
“doubleunderscore”)methods.
Whenwewriteageneratorfunction,Pythonautomaticallytransformsitintoanobjectthat
isverysimilartoaniterator,andwhenwecallnext(generator),thatcallistransformed
ingenerator.__next__().Let’srevisitthepreviousexampleaboutgeneratingsquares:
first.n.squares.manual.method.py
defget_squares_gen(n):
forxinrange(n):
yieldx**2
squares=get_squares_gen(3)
print(squares.__next__())#prints:0
print(squares.__next__())#prints:1
print(squares.__next__())#prints:4
#thefollowingraisesStopIteration,thegeneratorisexhausted,
#anyfurthercalltonextwillkeepraisingStopIteration
print(squares.__next__())
Theresultisexactlyasthepreviousexample,onlythistimeinsteadofusingtheproxycall
next(squares),we’redirectlycallingsquares.__next__().
Generatorobjectshavealsothreeothermethodsthatallowcontrollingtheirbehavior:
send,throw,andclose.sendallowsustocommunicateavaluebacktothegenerator
object,whilethrowandcloserespectivelyallowraisinganexceptionwithinthegenerator
andclosingit.TheiruseisquiteadvancedandIwon’tbecoveringthemhereindetail,but
Iwanttospendafewwordsatleastaboutsend,withasimpleexample.
Takealookatthefollowingcode:
gen.send.preparation.py
defcounter(start=0):
n=start
whileTrue:
yieldn
n+=1
WOW! eBook
www.wowebook.org
c=counter()
print(next(c))#prints:0
print(next(c))#prints:1
print(next(c))#prints:2
Theprecedingiteratorcreatesageneratorobjectthatwillrunforever.Youcankeep
callingit,itwillneverstop.Alternatively,youcanputitinaforloop,forexample,forn
incounter():...anditwillgoonforeveraswell.
Now,whatifyouwantedtostopitatsomepoint?Onesolutionistouseavariableto
controlthewhileloop.Somethinglikethis:
gen.send.preparation.stop.py
stop=False
defcounter(start=0):
n=start
whilenotstop:
yieldn
n+=1
c=counter()
print(next(c))#prints:0
print(next(c))#prints:1
stop=True
print(next(c))#raisesStopIteration
Thiswilldoit.Westartwithstop=False,anduntilwechangeittoTrue,thegenerator
willjustkeepgoing,likebefore.ThemomentwechangestoptoTruethough,thewhile
loopwillexit,andthenextcallwillraiseaStopIterationexception.Thistrickworks,
butIdon’tlikeit.Wedependonanexternalvariable,andthiscanleadtoissues:whatif
anotherfunctionchangesthatstop?Moreover,thecodeisscattered.Inanutshell,this
isn’tgoodenough.
Wecanmakeitbetterbyusinggenerator.send().Whenwecallgenerator.send(),the
valuethatwefeedtosendwillbepassedintothegenerator,executionisresumed,andwe
canfetchitviatheyieldexpression.Thisisallverycomplicatedwhenexplainedwith
words,solet’sseeanexample:
gen.send.py
defcounter(start=0):
n=start
whileTrue:
result=yieldn#A
print(type(result),result)#B
ifresult=='Q':
break
n+=1
c=counter()
print(next(c))#C
print(c.send('Wow!'))#D
print(next(c))#E
print(c.send('Q'))#F
WOW! eBook
www.wowebook.org
Executionoftheprecedingcodeproducesthefollowing:
$pythongen.send.py
0
<class'str'>Wow!
1
<class'NoneType'>None
2
<class'str'>Q
Traceback(mostrecentcalllast):
File"gen.send.py",line14,in<module>
print(c.send('Q'))#F
StopIteration
Ithinkit’sworthgoingthroughthiscodelinebyline,likeifwewereexecutingit,andsee
ifwecanunderstandwhat’sgoingon.
Westartthegeneratorexecutionwithacalltonext(#C).Withinthegenerator,nissetto
thesamevalueofstart.Thewhileloopisentered,executionstops(#A)andn(0)is
yieldedbacktothecaller.0isprintedontheconsole.
Wethencallsend(#D),executionresumesandresultissetto'Wow!'(still#A),thenits
typeandvalueareprintedontheconsole(#B).resultisnot'Q',thereforenis
incrementedby1andexecutiongoesbacktothewhilecondition,which,beingTrue,
evaluatestoTrue(thatwasn’thardtoguess,right?).Anotherloopcyclebegins,execution
stopsagain(#A),andn(1)isyieldedbacktothecaller.1isprintedontheconsole.
Atthispoint,wecallnext(#E),executionisresumedagain(#A),andbecausewearenot
sendinganythingtothegeneratorexplicitly,Pythonbehavesexactlylikefunctionsthatare
notusingthereturnstatement:theyieldnexpression(#A)returnsNone.result
thereforeissettoNone,anditstypeandvalueareyetagainprintedontheconsole(#B).
Executioncontinues,resultisnot'Q'sonisincrementedby1,andwestartanotherloop
again.Executionstopsagain(#A)andn(2)isyieldedbacktothecaller.2isprintedonthe
console.
Andnowforthegrandfinale:wecallsendagain(#F),butthistimewepassin'Q',
thereforewhenexecutionisresumed,resultissetto'Q'(#A).Itstypeandvalueare
printedontheconsole(#B),andthenfinallytheifclauseevaluatestoTrueandthewhile
loopisstoppedbythebreakstatement.Thegeneratornaturallyterminatesandthismeans
aStopIterationexceptionisraised.Youcanseetheprintofitstracebackonthelastfew
linesprintedontheconsole.
Thisisnotatallsimpletounderstandatfirst,soifit’snotcleartoyou,don’tbe
discouraged.Youcankeepreadingonandthenyoucancomebacktothisexampleafter
sometime.
Usingsendallowsforinterestingpatterns,andit’sworthnotingthatsendcanonlybe
usedtoresumetheexecution,nottostartit.Onlynextstartstheexecutionofagenerator.
WOW! eBook
www.wowebook.org
Theyieldfromexpression
Anotherinterestingconstructistheyieldfromexpression.Thisexpressionallowsyouto
yieldvaluesfromasubiterator.Itsuseallowsforquiteadvancedpatterns,solet’sjustsee
averyquickexampleofit:
gen.yield.for.py
defprint_squares(start,end):
forninrange(start,end):
yieldn**2
forninprint_squares(2,5):
print(n)
Thepreviouscodeprintsthenumbers4,9,16ontheconsole(onseparatelines).Bynow,I
expectyoutobeabletounderstanditbyyourself,butlet’squicklyrecapwhathappens.
Theforloopoutsidethefunctiongetsaniteratorfromprint_squares(2,5)andcalls
nextonituntiliterationisover.Everytimethegeneratoriscalled,executionissuspended
(andlaterresumed)onyieldn**2,whichreturnsthesquareofthecurrentn.
Let’sseehowwecantransformthiscodebenefitingfromtheyieldfromexpression:
gen.yield.from.py
defprint_squares(start,end):
yieldfrom(n**2forninrange(start,end))
forninprint_squares(2,5):
print(n)
Thiscodeproducesthesameresult,butasyoucanseetheyieldfromisactuallyrunning
asubiterator(n**2…).Theyieldfromexpressionreturnstothecallereachvaluethe
subiteratorisproducing.It’sshorteranditreadsbetter.
WOW! eBook
www.wowebook.org
Generatorexpressions
Let’snowtalkabouttheothertechniquestogeneratevaluesoneatatime.
Thesyntaxisexactlythesameaslistcomprehensions,only,insteadofwrappingthe
comprehensionwithsquarebrackets,youwrapitwithroundbraces.Thatiscalleda
generatorexpression.
Ingeneral,generatorexpressionsbehavelikeequivalentlistcomprehensions,butthereis
oneveryimportantthingtoremember:generatorsallowforoneiterationonly,thenthey
willbeexhausted.Let’sseeanexample:
generator.expressions.py
>>>cubes=[k**3forkinrange(10)]#regularlist
>>>cubes
[0,1,8,27,64,125,216,343,512,729]
>>>type(cubes)
<class'list'>
>>>cubes_gen=(k**3forkinrange(10))#createasgenerator
>>>cubes_gen
<generatorobject<genexpr>at0x7ff26b5db990>
>>>type(cubes_gen)
<class'generator'>
>>>list(cubes_gen)#thiswillexhaustthegenerator
[0,1,8,27,64,125,216,343,512,729]
>>>list(cubes_gen)#nothingmoretogive
[]
Lookatthelineinwhichthegeneratorexpressioniscreatedandassignedthename
cubes_gen.Youcanseeit’sageneratorobject.Inordertoseeitselements,wecanusea
forloop,amanualsetofcallstonext,orsimply,feedittoalistconstructor,whichis
whatIdid.
Noticehow,oncethegeneratorhasbeenexhausted,thereisnowaytorecoverthesame
elementsfromitagain.Weneedtorecreateit,ifwewanttouseitfromscratchagain.
Inthenextfewexamples,let’sseehowtoreproducemapandfilterusinggenerator
expressions:
gen.map.py
defadder(*n):
returnsum(n)
s1=sum(map(lambdan:adder(*n),zip(range(100),range(1,101))))
s2=sum(adder(*n)forninzip(range(100),range(1,101)))
Inthepreviousexample,s1ands2areexactlythesame:theyarethesumofadder(0,
1),adder(1,2),adder(2,3),andsoon,whichtranslatestosum(1,3,5,...).The
syntaxisdifferentthough,Ifindthegeneratorexpressiontobemuchmorereadable:
gen.filter.py
cubes=[x**3forxinrange(10)]
odd_cubes1=filter(lambdacube:cube%2,cubes)
WOW! eBook
www.wowebook.org
odd_cubes2=(cubeforcubeincubesifcube%2)
Inthepreviousexample,odd_cubes1andodd_cubes2arethesame:theygeneratea
sequenceofoddcubes.Yetagain,Ipreferthegeneratorsyntax.Thisshouldbeevident
whenthingsgetalittlemorecomplicated:
gen.map.filter.py
N=20
cubes1=map(
lambdan:(n,n**3),
filter(lambdan:n%3==0orn%5==0,range(N))
)
cubes2=(
(n,n**3)forninrange(N)ifn%3==0orn%5==0)
Theprecedingcodecreatestogeneratorscubes1andcubes2.Theyareexactlythesame,
andreturn2-tuples(n,
)whennisamultipleof3or5.
Ifyouprintthelist(cubes1),youget:[(0,0),(3,27),(5,125),(6,216),(9,
729),(10,1000),(12,1728),(15,3375),(18,5832)].
Seehowmuchbetterthegeneratorexpressionreads?Itmaybedebatablewhenthingsare
verysimple,butassoonasyoustartnestingfunctionsabit,likewedidinthisexample,
thesuperiorityofthegeneratorsyntaxisevident.Shorter,simpler,moreelegant.
Now,letmeaskyouaquestion:whatisthedifferencebetweenthefollowinglinesof
code?
sum.example.py
s1=sum([n**2forninrange(10**6)])
s2=sum((n**2forninrange(10**6)))
s3=sum(n**2forninrange(10**6))
Strictlyspeaking,theyallproducethesamesum.Theexpressionstogets2ands3are
exactlythesamebecausethebracesins2areredundant.Theyarebothgenerator
expressionsinsidethesumfunction.Theexpressiontogets1isdifferentthough.Inside
sum,wefindalistcomprehension.Thismeansthatinordertocalculates1,thesum
functionhastocallnextonalist,amilliontimes.
Doyouseewherewe’reloosingtimeandmemory?Beforesumcanstartcallingnexton
thatlist,thelistneedstohavebeencreated,whichisawasteoftimeandspace.It’smuch
betterforsumtocallnextonasimplegeneratorexpression.Thereisnoneedtohaveall
thenumbersfromrange(10**6)storedinalist.
So,watchoutforextraparentheseswhenyouwriteyourexpressions:sometimesit’seasy
toskiponthesedetails,whichmakesourcodemuchdifferent.Don’tbelieveme?
sum.example.2.py
s=sum([n**2forninrange(10**8)])#thisiskilled
#s=sum(n**2forninrange(10**8))#thissucceeds
print(s)
WOW! eBook
www.wowebook.org
Tryrunningtheprecedingexample.IfIrunthefirstline,thisiswhatIget:
$pythonsum.example.2.py
Killed
Ontheotherhand,ifIcommentoutthefirstline,anduncommentthesecondone,thisis
theresult:
$pythonsum.example.2.py
333333328333333350000000
Sweetgeneratorexpressions.Thedifferencebetweenthetwolinesisthatinthefirstone,a
listwiththesquaresofthefirsthundredmillionnumbersmustbemadebeforebeingable
tosumthemup.Thatlistishuge,andwerunoutofmemory(atleast,myboxdid,ifyours
doesn’ttryabiggernumber),thereforePythonkillstheprocessforus.Sadface.
Butwhenweremovethesquarebrackets,wedon’tmakealistanymore.Thesum
functionreceives0,1,4,9,andsoonuntilthelastone,andsumsthemup.Noproblems,
happyface.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Someperformanceconsiderations
So,we’veseenthatwehavemanydifferentwaystoachievethesameresult.Wecanuse
anycombinationofmap,zip,filter,orchoosetogowithacomprehension,ormaybe
choosetouseagenerator,eitherfunctionorexpression.Wemayevendecidetogowith
forloops:whenthelogictoapplytoeachrunningparameterisn’tsimple,theymaybethe
bestoption.
Otherthanreadabilityconcernsthough,let’stalkaboutperformances.Whenitcomesto
performances,usuallytherearetwofactorswhichplayamajorrole:spaceandtime.
Spacemeansthesizeofthememorythatadatastructureisgoingtotakeup.Thebestway
tochooseistoaskyourselfifyoureallyneedalist(ortuple)orifasimplegenerator
functionwouldworkaswell.Iftheanswerisyes,gowiththegenerator,it’llsavealotof
space.Samegoeswithfunctions:ifyoudon’tactuallyneedthemtoreturnalistortuple,
thenyoucantransformthemingeneratorfunctionsaswell.
Sometimes,youwillhavetouselists(ortuples),forexampletherearealgorithmsthat
scansequencesusingmultiplepointersormaybetheyrunoverthesequencemorethan
once.Ageneratorfunction(orexpression)canbeiteratedoveronlyonceandthenit’s
exhausted,sointhesesituations,itwouldn’tbetherightchoice.
Timeisabitharderthanspacebecauseitdependsonmorevariablesandthereforeitisn’t
possibletostatethatXisfasterthanYwithabsolutecertaintyforallcases.However,
basedontestsrunonPythontoday,wecansaythatmapcallscanbetwiceasfastas
equivalentforloops,andlistcomprehensionscanbe(alwaysgenerallyspeaking)even
fasterthanequivalentmapcalls.
Inordertofullyappreciatethereasonbehindthesestatements,weneedtounderstandhow
Pythonworks,andthisisabitoutsidethescopeofthisbook,forit’stootechnicalin
detail.Let’sjustsaythatmapandlistcomprehensionsrunatClanguagespeedwithinthe
interpreter,whileaPythonforloopisrunasPythonbytecodewithinthePythonVirtual
Machine,whichisoftenmuchslower.
Note
ThereareseveraldifferentimplementationsofPython.Theoriginalone,andstillthemost
commonone,istheonewritteninC.Cisoneofthemostpowerfulandpopular
programminglanguagesstillusedtoday.
TheseclaimsImadecomefrombooksandarticlesthatyoucanfindontheWeb,buthow
aboutwedoasmallexerciseandtrytofindoutforourselves?Iwillwriteasmallpieceof
codethatcollectstheresultsofdivmod(a,b)foracertainsetofintegerpairs(a,b).I
willusethetimefunctionfromthetimemoduletocalculatetheelapsedtimeofthe
operationsthatIwillperform.Let’sgo!
performances.py
fromtimeimporttime
mx=5500#thisisthemaxIcouldreachwithmycomputer…
WOW! eBook
www.wowebook.org
t=time()#starttimefortheforloop
dmloop=[]
forainrange(1,mx):
forbinrange(a,mx):
dmloop.append(divmod(a,b))
print('forloop:{:.4f}s'.format(time()-t))#elapsedtime
t=time()#starttimeforthelistcomprehension
dmlist=[
divmod(a,b)forainrange(1,mx)forbinrange(a,mx)]
print('listcomprehension:{:.4f}s'.format(time()-t))
t=time()#starttimeforthegeneratorexpression
dmgen=list(
divmod(a,b)forainrange(1,mx)forbinrange(a,mx))
print('generatorexpression:{:.4f}s'.format(time()-t))
#verifycorrectnessofresultsandnumberofitemsineachlist
print(dmloop==dmlist==dmgen,len(dmloop))
Asyoucansee,we’recreatingthreelists:dmloop,dmlist,dmgen(divmod-forloop,
divmod-listcomprehension,divmod-generatorexpression).Westartwiththeslowest
option,theforloops.Thenwehavealistcomprehension,andfinallyagenerator
expression.Let’sseetheoutput:
$pythonperformances.py
forloop:4.3433s
listcomprehension:2.7238s
generatorexpression:3.1380s
True15122250
Thelistcomprehensionrunsin63%ofthetimetakenbytheforloop.That’s
impressive.Thegeneratorexpressioncamequiteclosetothat,withagood72%.The
reasonthegeneratorexpressionissloweristhatweneedtofeedittothelist()
constructorandthishasalittlebitmoreoverheadcomparedtoasheerlistcomprehension.
Iwouldnevergowithageneratorexpressioninasimilarcasethough,thereisnopointif
attheendwewantalist.Iwouldjustusealistcomprehension,andtheresultofthe
previousexampleprovesmeright.Ontheotherhand,ifIjusthadtodothosedivmod
calculationswithoutretainingtheresults,thenageneratorexpressionwouldbethewayto
gobecauseinsuchasituationalistcomprehensionwouldunnecessarilyconsumealotof
space.
So,torecap:generatorsareveryfastandallowyoutosaveonspace.Listcomprehensions
areingeneralevenfaster,butdon’tsaveonspace.PurePythonforloopsaretheslowest
option.Let’sseeasimilarexamplethatcomparesaforloopandamapcall:
performances.map.py
fromtimeimporttime
mx=2*10**7
t=time()
WOW! eBook
www.wowebook.org
absloop=[]
forninrange(mx):
absloop.append(abs(n))
print('forloop:{:.4f}s'.format(time()-t))
t=time()
abslist=[abs(n)forninrange(mx)]
print('listcomprehension:{:.4f}s'.format(time()-t))
t=time()
absmap=list(map(abs,range(mx)))
print('map:{:.4f}s'.format(time()-t))
print(absloop==abslist==absmap)
Thiscodeisconceptuallyverysimilartothepreviousexample.Theonlythingthathas
changedisthatwe’reapplyingtheabsfunctioninsteadofthedivmodone,andwehave
onlyoneloopinsteadoftwonestedones.Executiongivesthefollowingresult:
$pythonperformances.map.py
forloop:3.1283s
listcomprehension:1.3966s
map:1.2319s
True
Andmapwinstherace!AsItoldyoubefore,givingastatementofwhatisfasterthanwhat
isverytricky.Inthiscase,themapcallisfasterthanthelistcomprehension.
Apartfromthecasebycaselittledifferencesthough,it’squiteclearthattheforloop
optionistheslowestone,solet’sseewhatarethereasonswestillwanttouseit.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Don’toverdocomprehensionsand
generators
We’veseenhowpowerfullistcomprehensionsandgeneratorexpressionscanbe.Andthey
are,don’tgetmewrong,butthefeelingthatIhavewhenIdealwiththemisthattheir
complexitygrowsexponentially.Themoreyoutrytodowithinasinglecomprehensionor
ageneratorexpression,theharderitbecomestoread,understand,andthereforeto
maintainorchange.
OpenaPythonconsoleandtypeinimportthis,let’sreadtheZenofPythonagain,in
particular,thereareafewlinesthatIthinkareveryimportanttokeepinmind:
>>>importthis
TheZenofPython,byTimPeters
Beautifulisbetterthanugly.
Explicitisbetterthanimplicit.#
Simpleisbetterthancomplex.#
Complexisbetterthancomplicated.
Flatisbetterthannested.
Sparseisbetterthandense.
Readabilitycounts.#
Specialcasesaren'tspecialenoughtobreaktherules.
Althoughpracticalitybeatspurity.
Errorsshouldneverpasssilently.
Unlessexplicitlysilenced.
Inthefaceofambiguity,refusethetemptationtoguess.
Thereshouldbeone--andpreferablyonlyone--obviouswaytodoit.
Althoughthatwaymaynotbeobviousatfirstunlessyou'reDutch.
Nowisbetterthannever.
Althoughneverisoftenbetterthan*right*now.
Iftheimplementationishardtoexplain,it'sabadidea.#
Iftheimplementationiseasytoexplain,itmaybeagoodidea.
Namespacesareonehonkinggreatidea—let'sdomoreofthose!
Ihaveputacommentsignontherightofthemainfocuspointshere.Comprehensionsand
generatorexpressionsbecomehardtoread,moreimplicitthanexplicit,complex,andthey
canbehardtoexplain.Sometimesyouhavetobreakthemapartusingtheinside-out
technique,tounderstandwhytheyproducetheresulttheyproduce.
Togiveyouanexample,let’stalkabitmoreaboutPythagoreantriples.Justtoremind
you,aPythagoreantripleisatupleofpositiveintegers(a,b,c)suchthat
.
Wesawearlierinthischapterhowtocalculatethem,butwediditinaveryinefficient
waybecausewewerescanningallpairsofnumbersbelowacertainthreshold,calculating
thehypotenuse,andfilteringoutthosethatwerenotproducingatriple.
AbetterwaytogetalistofPythagoreantriplesistodirectlygeneratethem.Thereare
manydifferentformulastodothisandwe’lluseoneofthem:theEuclideanformula.
WOW! eBook
www.wowebook.org
Thisformulasaysthatanytriple(a,b,c),where
,b=2mn,
,with
mandnpositiveintegerssuchthatm>n,isaPythagoreantriple.Forexample,whenm=
2andn=1,wefindthesmallesttriple:(3,4,5).
Thereisonecatchthough:considerthetriple(6,8,10),thatisjustlike(3,4,5)withall
,but
thenumbersmultipliedby2.ThistripleisdefinitelyPythagorean,since
wecanderiveitfrom(3,4,5)simplybymultiplyingeachofitselementsby2.Samegoes
for(9,12,15),(12,16,20),andingeneralforallthetriplesthatwecanwriteas(3k,4k,
5k),withkbeingapositiveintegergreaterthan1.
Atriplethatcannotbeobtainedbymultiplyingtheelementsofanotheronebysomefactor
k,iscalledprimitive.Anotherwayofstatingthisis:ifthethreeelementsofatripleare
coprime,thenthetripleisprimitive.Twonumbersarecoprimewhentheydon’tshareany
primefactoramongsttheirdivisors,thatis,theirgreatestcommondivisor(GCD)is1.
Forexample,3and5arecoprime,while3and6arenot,becausetheyarebothdivisible
by3.
So,theEuclideanformulatellsusthatifmandnarecoprime,andm–nisodd,thetriple
theygenerateisprimitive.Inthefollowingexample,wewillwriteageneratorexpression
tocalculatealltheprimitivePythagoreantripleswhosehypotenuse(c)islessthanorequal
tosomeintegerN.Thismeanswewantalltriplesforwhich
,whichmeanswecanapproximatethecalculationwith
formulalookslikethis:
anupperboundof
.Whennis1,the
.
So,torecap:mmustbegreaterthann,theymustalsobecoprime,andtheirdifferencemnmustbeodd.Moreover,inordertoavoiduselesscalculationswe’llputtheupperbound
formatfloor(sqrt(N))+1.
Note
Thefunctionfloorforarealnumberxgivesthemaximumintegernsuchthatn<x,for
example,floor(3.8)=3,floor(13.1)=13.Takingthefloor(sqrt(N))+1meanstakingthe
integerpartofthesquarerootofNandaddingaminimalmarginjusttomakesurewe
don’tmissoutanynumber.
Let’sputallofthisintocode,stepbystep.Let’sstartbywritingasimplegcdfunctionthat
usesEuclid’salgorithm:
functions.py
defgcd(a,b):
"""CalculatetheGreatestCommonDivisorof(a,b)."""
whileb!=0:
a,b=b,a%b
returna
TheexplanationofEuclid’salgorithmisavailableontheWeb,soIwon’tspendanytime
heretalkingaboutit;weneedtoconcentrateonthegeneratorexpression.Thenextstepis
tousetheknowledgewegatheredbeforetogeneratealistofprimitivePythagorean
WOW! eBook
www.wowebook.org
triples:
pythagorean.triple.generation.py
fromfunctionsimportgcd
N=50
triples=sorted(#1
((a,b,c)fora,b,cin(#2
((m**2-n**2),(2*m*n),(m**2+n**2))#3
forminrange(1,int(N**.5)+1)#4
forninrange(1,m)#5
if(m-n)%2andgcd(m,n)==1#6
)ifc<=N),key=lambda*triple:sum(*triple)#7
)
print(triples)
Thereyougo.It’snoteasytoread,solet’sgothroughitlinebyline.At#3,westarta
generatorexpressionthatiscreatingtriples.Youcanseefrom#4and#5thatwe’relooping
onmin[1,M]withMbeingtheintegerpartofsqrt(N),plus1.Ontheotherhand,nloops
within[1,m),torespectthem>nrule.WorthnotinghowIcalculatedsqrt(N),thatis,
N**.5,whichisjustanotherwaytodoitthatIwantedtoshowyou.
At#6,youcanseethefilteringconditionstomakethetriplesprimitive:(m-n)%2
evaluatestoTruewhen(m-n)isodd,andgcd(m,n)==1meansmandnarecoprime.
Withtheseinplace,weknowthetripleswillbeprimitive.Thistakescareoftheinnermost
generatorexpression.Theoutermostonestartsat#2,andfinishesat#7.Wetakethetriples
(a,b,c)in(…innermostgenerator…)suchthatc<=N.Thisisnecessarybecause
isthelowestupperboundthatwecanapply,butitdoesn’tguaranteethatcwill
actuallybelessthanorequaltoN.
Finally,at#1weapplysorting,topresentthelistinorder.At#7,aftertheoutermost
generatorexpressionisclosed,youcanseethatwespecifythesortingkeytobethesuma
+b+c.Thisisjustmypersonalpreference,thereisnomathematicalreasonbehindit.
So,whatdoyouthink?Wasitstraightforwardtoread?Idon’tthinkso.Andbelieveme,
thisisstillasimpleexample;Ihaveseenexpressionswaymorecomplicatedthanthisone.
Unfortunatelysomeprogrammersthinkthatwritingcodelikethisiscool,thatit’ssome
sortofdemonstrationoftheirsuperiorintellectualpowers,oftheirabilitytoquicklyread
anddigestintricatecode.
Withinaprofessionalenvironmentthough,Ifindmyselfhavingmuchmorerespectfor
thosewhowriteefficient,cleancode,andmanagetokeepegooutthedoor.Conversely,
thosewhodon’t,willproducelinesatwhichyouwillstareforalongtimewhileswearing
inthreelanguages(atleastthisiswhatIdo).
Now,let’sseeifwecanrewritethiscodeintosomethingeasiertoread:
pythagorean.triple.generation.for.py
fromfunctionsimportgcd
WOW! eBook
www.wowebook.org
defgen_triples(N):
forminrange(1,int(N**.5)+1):#1
forninrange(1,m):#2
if(m-n)%2andgcd(m,n)==1:#3
c=m**2+n**2#4
ifc<=N:#5
a=m**2-n**2#6
b=2*m*n#7
yield(a,b,c)#8
triples=sorted(
gen_triples(50),key=lambda*triple:sum(*triple))#9
print(triples)
Ifeelsomuchbetteralready.Let’sgothroughthiscodeaswell,linebyline.You’llsee
howeasieritistounderstand.
Westartloopingat#1and#2,inexactlythesamewaywewereloopingintheprevious
example.Online#3,wehavethefilteringforprimitivetriples.Online#4,wedeviatea
bitfromwhatweweredoingbefore:wecalculatec,andonline#5,wefilteroncbeing
lessthanorequaltoN.Onlywhencsatisfiesthatcondition,wecalculateaandb,and
yieldtheresultingtuple.It’salwaysgoodtodelayallcalculationsforasmuchaspossible
sothatwedon’twastetime,incaseeventuallywehavetodiscardthoseresults.
Onthelastline,beforeprintingtheresult,weapplysortingwiththesamekeywewere
usinginthegeneratorexpressionexample.
Ihopeyouagree,thisexampleiseasiertounderstand.AndIpromiseyou,ifyouhaveto
modifythecodeoneday,you’llfindthatmodifyingthisoneiseasy,whiletomodifythe
otherversionwilltakemuchlonger(anditwillbemoreerrorprone).
Bothexamples,whenrun,printthefollowing:
$pythonpythagorean.triple.generation.py
[(3,4,5),(5,12,13),(15,8,17),(7,24,25),(21,20,29),(35,12,
37),(9,40,41)]
Themoralofthestoryis,tryandusecomprehensionsandgeneratorexpressionsasmuch
asyoucan,butifthecodestartstobecomplicatedtomodifyortoread,youmaywantto
refactorintosomethingmorereadable.Thereisnothingwrongwiththis.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Namelocalization
Nowthatwearefamiliarwithalltypesofcomprehensionsandgeneratorexpression,let’s
talkaboutnamelocalizationwithinthem.Python3.*localizesloopvariablesinallfour
formsofcomprehensions:list,dict,set,andgeneratorexpressions.Thisbehavioris
thereforedifferentfromthatoftheforloop.Let’sseeasimpleexampletoshowallthe
cases:
scopes.py
A=100
ex1=[AforAinrange(5)]
print(A)#prints:100
ex2=list(AforAinrange(5))
print(A)#prints:100
ex3=dict((A,2*A)forAinrange(5))
print(A)#prints:100
ex4=set(AforAinrange(5))
print(A)#prints:100
s=0
forAinrange(5):
s+=A
print(A)#prints:4
Intheprecedingcode,wedeclareaglobalnameA=100,andthenweexercisethefour
comprehensions:list,generatorexpression,dictionary,andset.Noneofthemalterthe
globalnameA.Conversely,youcanseeattheendthattheforloopmodifiesit.Thelast
printstatementprints4.
Let’sseewhathappensifAwasn’tthere:
scopes.noglobal.py
ex1=[AforAinrange(5)]
print(A)#breaks:NameError:name'A'isnotdefined
Theprecedingcodewouldworkthesamewithanyofthefourtypesofcomprehensions.
Afterwerunthefirstline,Aisnotdefinedintheglobalnamespace.
Onceagain,theforloopbehavesdifferently:
scopes.for.py
s=0
forAinrange(5):
s+=A
print(A)#prints:4
print(globals())
Theprecedingcodeshowsthatafteraforloop,iftheloopvariablewasn’tdefinedbefore
it,wecanfinditintheglobalframe.Tomakesureofit,let’stakeapeekatitbycalling
WOW! eBook
www.wowebook.org
theglobals()built-infunction:
$pythonscopes.for.py
4
{'__spec__':None,'__name__':'__main__','s':10,'A':4,'__doc__':
None,'__cached__':None,'__package__':None,'__file__':'scopes.for.py',
'__loader__':<_frozen_importlib.SourceFileLoaderobjectat
0x7f05a5a183c8>,'__builtins__':<module'builtins'(built-in)>}
Togetherwithalotofotherboilerplatestuff,wecanspot'A':4.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Generationbehaviorinbuilt-ins
Amongstthebuilt-intypes,thegenerationbehaviorisnowquitecommon.Thisisamajor
differencebetweenPython2andPython3.Alotoffunctionssuchasmap,zip,and
filterhavebeentransformedsothattheyreturnobjectsthatbehavelikeiterables.The
ideabehindthischangeisthatifyouneedtomakealistofthoseresultsyoucanalways
wrapthecallinalist()class,andyou’redone.Ontheotherhand,ifyoujustneedto
iterateandwanttokeeptheimpactonmemoryaslightaspossible,youcanusethose
functionssafely.
Anothernotableexampleistherangefunction.InPython2itreturnsalist,andthereis
anotherfunctioncalledxrangethatreturnsanobjectthatyoucaniterateon,which
generatesthenumbersonthefly.InPython3thisfunctionhasgone,andrangenow
behaveslikeit.
Butthisconceptingeneralisnowquitewidespread.Youcanfinditintheopen()
function,whichisusedtooperateonfileobjects(we’llseeitinoneofthenextchapters),
butalsoinenumerate,inthedictionarykeys,values,anditemsmethods,andseveral
otherplaces.
Itallmakessense:Python’saimistotryandreducethememoryfootprintbyavoiding
wastingspacewhereverispossible,especiallyinthosefunctionsandmethodsthatare
usedextensivelyinmostsituations.
Doyourememberatthebeginningofthischapter?Isaidthatitmakesmoresenseto
optimizetheperformancesofcodethathastodealwithalotofobjects,ratherthan
shavingoffafewmillisecondsfromafunctionthatwecalltwiceaday.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Onelastexample
Beforewepartfromthischapter,I’llshowyouasimpleproblemthatIsubmittedto
candidatesforaPythondeveloperroleinacompanyIusedtoworkfor.
Theproblemisthefollowing:giventhesequence01123581321…writeafunction
thatwouldreturnthetermsofthissequenceuptosomelimitN.
Ifyouhaven’trecognizedit,thatistheFibonaccisequence,whichisdefinedasF(0)=0,
F(1)=1and,foranyn>1,F(n)=F(n-1)+F(n-2).Thissequenceisexcellenttotest
knowledgeaboutrecursion,memoizationtechniquesandothertechnicaldetails,butinthis
caseitwasagoodopportunitytocheckwhetherthecandidateknewaboutgenerators(and
toomanysocalledPythoncodersdidn’t,whenIwasinterviewingthem).
Let’sstartfromarudimentaryversionofafunction,andthenimproveonit:
fibonacci.first.py
deffibonacci(N):
"""ReturnallfibonaccinumbersuptoN."""
result=[0]
next_n=1
whilenext_n<=N:
result.append(next_n)
next_n=sum(result[-2:])
returnresult
print(fibonacci(0))#[0]
print(fibonacci(1))#[0,1,1]
print(fibonacci(50))#[0,1,1,2,3,5,8,13,21,34]
Fromthetop:wesetuptheresultlisttoastartingvalueof[0].Thenwestartthe
iterationfromthenextelement(next_n),whichis1.Whilethenextelementisnotgreater
thanN,wekeepappendingittothelistandcalculatingthenext.Wecalculatethenext
elementbytakingasliceofthelasttwoelementsintheresultlistandpassingittothe
sumfunction.Addsomeprintstatementshereandthereifthisisnotcleartoyou,butby
nowIwouldexpectitnottobeanissue.
WhentheconditionofthewhileloopevaluatestoFalse,weexittheloopandreturn
result.Youcanseetheresultofthoseprintstatementsinthecommentsnexttoeachof
them.
Atthispoint,Iwouldaskthecandidatethefollowingquestion:“WhatifIjustwantedto
iterateoverthosenumbers?”Agoodcandidatewouldthenchangethecodelikethenext
listing(anexcellentcandidatewouldhavestartedwithit!):
fibonacci.second.py
deffibonacci(N):
"""ReturnallfibonaccinumbersuptoN."""
yield0
ifN==0:
return
WOW! eBook
www.wowebook.org
a=0
b=1
whileb<=N:
yieldb
a,b=b,a+b
print(list(fibonacci(0)))#[0]
print(list(fibonacci(1)))#[0,1,1]
print(list(fibonacci(50)))#[0,1,1,2,3,5,8,13,21,34]
ThisisactuallyoneofthesolutionsIwasgiven.Idon’tknowwhyIkeptit,butI’mgladI
didsoIcanshowittoyou.Now,thefibonaccifunctionisageneratorfunction.Firstwe
yield0,thenifNis0wereturn(thiswillcauseaStopIterationexceptiontoberaised).If
that’snotthecase,westartiterating,yieldingbateveryloopcycle,andthenupdatinga
andb.Allweneedinordertobeabletoproducethenextelementofthesequenceisthe
pasttwo:aandb,respectively.
Thiscodeismuchbetter,hasalightermemoryfootprintandallwehavetodotogetalist
ofFibonaccinumbersistowrapthecallwithlist(),asusual.
Butwhataboutelegance?Icannotleavethecodelikethat.Itwasdecentforaninterview,
wherethefocusismoreonfunctionalitythanelegance,buthereI’dliketoshowyoua
nicerversion:
fibonacci.elegant.py
deffibonacci(N):
"""ReturnallfibonaccinumbersuptoN."""
a,b=0,1
whilea<=N:
yielda
a,b=b,a+b
Muchbetter.Thewholebodyofthefunctionisfourlines,fiveifyoucountthedocstring.
Noticehowinthiscaseusingtupleassignment(a,b=0,1anda,b=b,a+b)
helpsinmakingthecodeshorter,andmorereadable.It’soneofthefeaturesofPythonI
likealot.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,weexploredtheconceptofiterationandgenerationabitmoredeeply.We
sawthemap,zipandfilterfunctionsquiteindetail,andhowtousethemasan
alternativetoaregularforloopapproach.
Thenwesawtheconceptofcomprehensions,forlists,dictionaries,andsets.Wesawtheir
syntaxandhowtousethemasanalternativetoboththeclassicforloopapproachand
alsototheuseofmap,zip,andfilterfunctions.
Finally,wetalkedabouttheconceptofgeneration,intwoforms:generatorfunctionsand
expressions.Welearnedhowtosavetimeandspacebyusinggenerationtechniquesand
sawhowtheycanmakepossiblewhatwouldn’tnormallybeifweusedaconventional
approachbasedonlists.
Wetalkedaboutperformances,andsawthatforloopsarelastintermsofspeed,butthey
providethebestreadabilityandflexibilitytochange.Ontheotherhand,functionssuchas
mapandfiltercanbemuchfaster,andcomprehensionsmaybeevenbetter.
Thecomplexityofthecodewrittenusingthesetechniquesgrowsexponentiallyso,in
ordertofavorreadabilityandeaseofmaintainability,westillneedtousetheclassicfor
loopapproachattimes.Anotherdifferenceisinthenamelocalization,wheretheforloop
behavesdifferentlyfromallothertypesofcomprehensions.
Thenextchapterwillbeallaboutobjectsandclasses.Structurallysimilartothisone,in
thatwewon’texploremanydifferentsubjects,rather,justafewofthem,butwe’lltryto
divealittlebitmoredeeply.
Makesureyouunderstandwelltheconceptsofthischapterbeforejumpingtothenext
one.We’rebuildingawallbrickbybrick,andifthefoundationisnotsolid,wewon’tget
veryfar.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter6.AdvancedConcepts–OOP,
Decorators,andIterators
“Laclassenonèacqua.(Classwillout)” —Italiansaying
Icouldprobablywriteasmallbookaboutobject-orientedprogramming(referredtoas
OOPhenceforth)andclasses.Inthischapter,I’mfacingthehardchallengeoffindingthe
balancebetweenbreadthanddepth.Therearesimplytoomanythingstotell,andthere’s
plentyofthemthatwouldtakemorethanthiswholechapterifIdescribedthemalonein
depth.Therefore,IwilltrytogiveyouwhatIthinkisagoodpanoramicviewofthe
fundamentals,plusafewthingsthatmaycomeinhandyinthenextchapters.Python’s
officialdocumentationwillhelpinfillingthegaps.
We’regoingtoexplorethreeimportantconceptsinthischapter:decorators,OOP,and
iterators.
WOW! eBook
www.wowebook.org
Decorators
Inthepreviouschapter,Imeasuredtheexecutiontimeofvariousexpressions.Ifyou
recall,Ihadtoinitializeavariabletothestarttime,andsubtractitfromthecurrenttime
afterexecutioninordertocalculatetheelapsedtime.Ialsoprinteditontheconsoleafter
eachmeasurement.Thatwasverytedious.
Everytimeyoufindyourselfrepeatingthings,analarmbellshouldgooff.Canyouput
thatcodeinafunctionandavoidrepetition?Theanswermostofthetimeisyes,solet’s
lookatanexample.
decorators/time.measure.start.py
fromtimeimportsleep,time
deff():
sleep(.3)
defg():
sleep(.5)
t=time()
f()
print('ftook:',time()-t)#ftook:0.3003859519958496
t=time()
g()
print('gtook:',time()-t)#gtook:0.5005719661712646
Intheprecedingcode,Idefinedtwofunctions,fandg,whichdonothingbutsleep(by0.3
and0.5secondsrespectively).Iusedthesleepfunctiontosuspendtheexecutionforthe
desiredamountoftime.Ialsohighlightedhowwecalculatethetimeelapsedbysettingt
tothecurrenttimeandthensubtractingitwhenthetaskisdone.Youcanseethatthe
measureisprettyaccurate.
Now,howdoweavoidrepeatingthatcodeandthosecalculations?Onefirstpotential
approachcouldbethefollowing:
decorators/time.measure.dry.py
fromtimeimportsleep,time
deff():
sleep(.3)
defg():
sleep(.5)
defmeasure(func):
t=time()
func()
print(func.__name__,'took:',time()-t)
measure(f)#ftook:0.30041074752807617
WOW! eBook
www.wowebook.org
measure(g)#gtook:0.5006198883056641
Ah,muchbetternow.Thewholetimingmechanismhasbeenencapsulatedintoafunction
sowedon’trepeatcode.Weprintthefunctionnamedynamicallyandit’seasyenoughto
code.Whatifweneedtopassargumentstothefunctionwemeasure?Thiscodewouldget
justabitmorecomplicated,solet’sseeanexample.
decorators/time.measure.arguments.py
fromtimeimportsleep,time
deff(sleep_time=0.1):
sleep(sleep_time)
defmeasure(func,*args,**kwargs):
t=time()
func(*args,**kwargs)
print(func.__name__,'took:',time()-t)
measure(f,sleep_time=0.3)#ftook:0.3004162311553955
measure(f,0.2)#ftook:0.20028162002563477
Now,fisexpectingtobefedsleep_time(withadefaultvalueof0.1).Ialsohadto
changethemeasurefunctionsothatitisnowacceptingafunction,anyvariablepositional
arguments,andanyvariablekeywordarguments.Inthisway,whateverwecallmeasure
with,weredirectthoseargumentstothecalltofwedoinside.
Thisisverygood,butwecanpushitalittlebitfurther.Let’ssaywewanttosomehow
havethattimingbehaviorbuilt-inintheffunction,sothatwecouldjustcallitandhave
thatmeasuretaken.Here’showwecoulddoit:
decorators/time.measure.deco1.py
fromtimeimportsleep,time
deff(sleep_time=0.1):
sleep(sleep_time)
defmeasure(func):
defwrapper(*args,**kwargs):
t=time()
func(*args,**kwargs)
print(func.__name__,'took:',time()-t)
returnwrapper
f=measure(f)#decorationpoint
f(0.2)#ftook:0.2002875804901123
f(sleep_time=0.3)#ftook:0.3003721237182617
print(f.__name__)#wrapper<-ouch!
Theprecedingcodeisprobablynotsostraightforward.Iconfessthat,eventoday,it
sometimesrequiresmesomeseriousconcentrationtounderstandsomedecorators,they
canbeprettynasty.Let’sseewhathappenshere.Themagicisinthedecorationpoint.
Webasicallyreassignfwithwhateverisreturnedbymeasurewhenwecallitwithfasan
WOW! eBook
www.wowebook.org
argument.Withinmeasure,wedefineanotherfunction,wrapper,andthenwereturnit.So,
theneteffectisthatafterthedecorationpoint,whenwecallf,we’reactuallycalling
wrapper.Sincethewrapperinsideiscallingfunc,whichisf,weareactuallyclosingthe
looplikethat.Ifyoudon’tbelieveme,takealookatthelastline.
wrapperisactually…awrapper.Ittakesvariableandpositionalarguments,andcallsf
withthem.Italsodoesthetimemeasurementtrickaroundthecall.
Thistechniqueiscalleddecoration,andmeasureis,atalleffects,adecorator.This
paradigmbecamesopopularandwidelyusedthatatsomepoint,Pythonaddedaspecial
syntaxforit(checkPEP318).Let’sexplorethreecases:onedecorator,twodecorators,
andonedecoratorthattakesarguments.
decorators/syntax.py
deffunc(arg1,arg2,...):
pass
func=decorator(func)
#isequivalenttothefollowing:
@decorator
deffunc(arg1,arg2,...):
pass
Basically,insteadofmanuallyreassigningthefunctiontowhatwasreturnedbythe
decorator,weprependthedefinitionofthefunctionwiththespecialsyntax
@decorator_name.
Wecanapplymultipledecoratorstothesamefunctioninthefollowingway:
decorators/syntax.py
deffunc(arg1,arg2,...):
pass
func=deco1(deco2(func))
#isequivalenttothefollowing:
@deco1
@deco2
deffunc(arg1,arg2,...):
pass
Whenapplyingmultipledecorators,payattentiontotheorder,shoulditmatter.Inthe
precedingexample,funcisdecoratedwithdeco2first,andtheresultisdecoratedwith
deco1.Agoodruleofthumbis:thecloserthedecoratortothefunction,thesooneritis
applied.
Somedecoratorscantakearguments.Thistechniqueisgenerallyusedtoproduceother
decorators.Let’slookatthesyntax,andthenwe’llseeanexampleofit.
decorators/syntax.py
deffunc(arg1,arg2,...):
WOW! eBook
www.wowebook.org
pass
func=decoarg(argA,argB)(func)
#isequivalenttothefollowing:
@decoarg(argA,argB)
deffunc(arg1,arg2,...):
pass
Asyoucansee,thiscaseisabitdifferent.Firstdecoargiscalledwiththegiven
arguments,andthenitsreturnvalue(theactualdecorator)iscalledwithfunc.BeforeI
giveyouanotherexample,let’sfixonethingthatisbotheringme.Idon’twanttolosethe
originalfunctionnameanddocstring(andtheotherattributesaswell,checkthe
documentationforthedetails)whenIdecorateit.Butbecauseinsideourdecoratorwe
returnwrapper,theoriginalattributesfromfuncarelostandfendsupbeingassignedthe
attributesofwrapper.Thereisaneasyfixforthatfromfunctools,awonderfulmodule
fromthePythonstandardlibrary.Iwillfixthelastexample,andIwillalsorewriteits
syntaxtousethe@operator.
decorators/time.measure.deco2.py
fromtimeimportsleep,time
fromfunctoolsimportwraps
defmeasure(func):
@wraps(func)
defwrapper(*args,**kwargs):
t=time()
func(*args,**kwargs)
print(func.__name__,'took:',time()-t)
returnwrapper
@measure
deff(sleep_time=0.1):
"""I'macat.Ilovetosleep!"""
sleep(sleep_time)
f(sleep_time=0.3)#ftook:0.30039525032043457
print(f.__name__,':',f.__doc__)
#f:I'macat.Ilovetosleep!
Nowwe’retalking!Asyoucansee,allweneedtodoistotellPythonthatwrapper
actuallywrapsfunc(bymeansofthewrapsfunction),andyoucanseethattheoriginal
nameanddocstringarenowmaintained.
Let’sseeanotherexample.Iwantadecoratorthatprintsanerrormessagewhentheresult
ofafunctionisgreaterthanathreshold.Iwillalsotakethisopportunitytoshowyouhow
toapplytwodecoratorsatonce.
decorators/two.decorators.py
fromtimeimportsleep,time
fromfunctoolsimportwraps
defmeasure(func):
WOW! eBook
www.wowebook.org
@wraps(func)
defwrapper(*args,**kwargs):
t=time()
result=func(*args,**kwargs)
print(func.__name__,'took:',time()-t)
returnresult
returnwrapper
defmax_result(func):
@wraps(func)
defwrapper(*args,**kwargs):
result=func(*args,**kwargs)
ifresult>100:
print('Resultistoobig({0}).Maxallowedis100.'
.format(result))
returnresult
returnwrapper
@measure
@max_result
defcube(n):
returnn**3
print(cube(2))
print(cube(5))
Tip
Takeyourtimeinstudyingtheprecedingexampleuntilyouaresureyouunderstandit
well.Ifyoudo,Idon’tthinkthereisanydecoratoryouwon’tbeabletowriteafterwards.
Ihadtoenhancethemeasuredecorator,sothatitswrappernowreturnstheresultofthe
calltofunc.Themax_resultdecoratordoesthataswell,butbeforereturning,itchecks
thatresultisnotgreaterthan100,whichisthemaximumallowed.
Idecoratedcubewithbothofthem.First,max_resultisapplied,thenmeasure.Running
thiscodeyieldsthisresult:
$pythontwo.decorators.py
cubetook:7.62939453125e-06#
8#
Resultistoobig(125).Maxallowedis100.
cubetook:1.1205673217773438e-05
125
Foryourconvenience,Iputa#totherightoftheresultsofthefirstcall:print(cube(2)).
Theresultis8,andthereforeitpassesthethresholdchecksilently.Therunningtimeis
measuredandprinted.Finally,weprinttheresult(8).
Onthesecondcall,theresultis125,sotheerrormessageisprinted,theresultreturned,
andthenit’stheturnofmeasure,whichprintstherunningtimeagain,andfinally,weprint
theresult(125).
HadIdecoratedthecubefunctionwiththesametwodecoratorsbutinadifferentorder,
theerrormessagewouldfollowthelinethatprintstherunningtime,insteadofpreceding
WOW! eBook
www.wowebook.org
it.
WOW! eBook
www.wowebook.org
Adecoratorfactory
Let’ssimplifythisexamplenow,goingbacktoasingledecorator:max_result.Iwantto
makeitsothatIcandecoratedifferentfunctionswithdifferentthresholds,andIdon’t
wanttowriteonedecoratorforeachthreshold.Let’samendmax_resultsothatitallows
ustodecoratefunctionsspecifyingthethresholddynamically.
decorators/decorators.factory.py
fromfunctoolsimportwraps
defmax_result(threshold):
defdecorator(func):
@wraps(func)
defwrapper(*args,**kwargs):
result=func(*args,**kwargs)
ifresult>threshold:
print(
'Resultistoobig({0}).Maxallowedis{1}.'
.format(result,threshold))
returnresult
returnwrapper
returndecorator
@max_result(75)
defcube(n):
returnn**3
print(cube(5))
Thisprecedingcodeshowsyouhowtowriteadecoratorfactory.Ifyourecall,
decoratingafunctionwithadecoratorthattakesargumentsisthesameaswritingfunc=
decorator(argA,argB)(func),sowhenwedecoratecubewithmax_result(75),we’re
doingcube=max_result(75)(cube).
Let’sgothroughwhathappens,stepbystep.Whenwecallmax_result(75),weenterits
body.Adecoratorfunctionisdefinedinside,whichtakesafunctionasitsonlyargument.
Insidethatfunction,theusualdecoratortrickisperformed.Wedefineawrapper,insideof
whichwechecktheresultoftheoriginalfunction’scall.Thebeautyofthisapproachis
thatfromtheinnermostlevel,wecanstillrefertobothfuncandthreshold,whichallows
ustosetthethresholddynamically.
wrapperreturnsresult,decoratorreturnswrapper,andmax_resultreturnsdecorator.
Thismeansthatourcallcube=max_result(75)(cube),actuallybecomescube=
decorator(cube).Notjustanydecoratorthough,butoneforwhichthresholdhasthe
value75.Thisisachievedbyamechanismcalledclosure,whichisoutsideofthescopeof
thischapterbutnonethelessveryinteresting,soImentioneditforyoutodosomeresearch
onit.
Runningthelastexampleproducesthefollowingresult:
$pythondecorators.factory.py
WOW! eBook
www.wowebook.org
Resultistoobig(125).Maxallowedis75.
125
Theprecedingcodeallowsmetousethemax_resultdecoratorwithdifferentthresholds
atmyownwill,likethis:
decorators/decorators.factory.py
@max_result(75)
defcube(n):
returnn**3
@max_result(100)
defsquare(n):
returnn**2
@max_result(1000)
defmultiply(a,b):
returna*b
Notethateverydecorationusesadifferentthresholdvalue.
DecoratorsareverypopularinPython.Theyareusedquiteoftenandtheysimplify(and
beautify,Idaresay)thecodealot.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Object-orientedprogramming
It’sbeenquitealongandhopefullynicejourneyand,bynow,weshouldbereadyto
exploreobject-orientedprogramming.I’llusethedefinitionfromKindler,E.;Krivy,I.
(2011).Object-OrientedSimulationofsystemswithsophisticatedcontrol.International
JournalofGeneralSystems,andadaptittoPython:
Object-orientedprogramming(OOP)isaprogrammingparadigmbasedonthe
conceptof“objects”,whicharedatastructuresthatcontaindata,intheformof
attributes,andcode,intheformoffunctionsknownasmethods.Adistinguishing
featureofobjectsisthatanobject’smethodcanaccessandoftenmodifythedata
attributesoftheobjectwithwhichtheyareassociated(objectshaveanotionof
“self”).InOOprogramming,computerprogramsaredesignedbymakingthemout
ofobjectsthatinteractwithoneanother.
Pythonhasfullsupportforthisparadigm.Actually,aswehavealreadysaid,everythingin
Pythonisanobject,sothisshowsthatOOPisnotjustsupportedbyPython,butit’spartof
itsverycore.
ThetwomainplayersinOOPareobjectsandclasses.Classesareusedtocreateobjects
(objectsareinstancesoftheclasseswithwhichtheywerecreated),sowecouldseethem
asinstancefactories.Whenobjectsarecreatedbyaclass,theyinherittheclassattributes
andmethods.Theyrepresentconcreteitemsintheprogram’sdomain.
WOW! eBook
www.wowebook.org
ThesimplestPythonclass
IwillstartwiththesimplestclassyoucouldeverwriteinPython.
oop/simplest.class.py
classSimplest():#whenempty,thebracesareoptional
pass
print(type(Simplest))#whattypeisthisobject?
simp=Simplest()#wecreateaninstanceofSimplest:simp
print(type(simp))#whattypeissimp?
#issimpaninstanceofSimplest?
print(type(simp)==Simplest)#There'sabetterwayforthis
Let’sruntheprecedingcodeandexplainitlinebyline:
$pythonoop/simplest.class.py
<class'type'>
<class'__main__.Simplest'>
True
TheSimplestclassIdefinedonlyhasthepassinstructionforitsbody,whichmeansit
doesn’thaveanycustomattributesormethods.Iwillprintitstype(__main__isthename
ofthescopeinwhichtop-levelcodeexecutes),andIamawarethat,inthecomment,I
wroteobjectinsteadofclass.Itturnsoutthat,asyoucanseebytheresultofthatprint,
classesareactuallyobjects.Tobeprecise,theyareinstancesoftype.Explainingthis
conceptwouldleadtoatalkaboutmetaclassesandmetaprogramming,advanced
conceptsthatrequireasolidgraspofthefundamentalstobeunderstoodandalasthisis
beyondthescopeofthischapter.Asusual,Imentionedittoleaveapointerforyou,for
whenyou’llbereadytodigdeeper.
Let’sgobacktotheexample:IusedSimplesttocreateaninstance,simp.Youcansee
thatthesyntaxtocreateaninstanceisthesameweusetocallafunction.
Thenweprintwhattypesimpbelongstoandweverifythatsimpisinfactaninstanceof
Simplest.I’llshowyouabetterwayofdoingthislateroninthechapter.
Uptonow,it’sallverysimple.WhathappenswhenwewriteclassClassName():pass,
though?Well,whatPythondoesiscreateaclassobjectandassignitaname.Thisisvery
similartowhathappenswhenwedeclareafunctionusingdef.
WOW! eBook
www.wowebook.org
Classandobjectnamespaces
Aftertheclassobjecthasbeencreated(whichusuallyhappenswhenthemoduleisfirst
imported),itbasicallyrepresentsanamespace.Wecancallthatclasstocreateits
instances.Eachinstanceinheritstheclassattributesandmethodsandisgivenitsown
namespace.Wealreadyknowthat,towalkanamespace,allweneedtodoistousethedot
(.)operator.
Let’slookatanotherexample:
oop/class.namespaces.py
classPerson():
species='Human'
print(Person.species)#Human
Person.alive=True#Addeddynamically!
print(Person.alive)#True
man=Person()
print(man.species)#Human(inherited)
print(man.alive)#True(inherited)
Person.alive=False
print(man.alive)#False(inherited)
man.name='Darth'
man.surname='Vader'
print(man.name,man.surname)#DarthVader
Intheprecedingexample,Ihavedefinedaclassattributecalledspecies.Anyvariable
definedinthebodyofaclassisanattributethatbelongstothatclass.Inthecode,Ihave
alsodefinedPerson.alive,whichisanotherclassattribute.Youcanseethatthereisno
restrictiononaccessingthatattributefromtheclass.Youcanseethatman,whichisan
instanceofPerson,inheritsbothofthem,andreflectstheminstantlywhentheychange.
manhasalsotwoattributeswhichbelongtoitsownnamespaceandthereforearecalled
instanceattributes:nameandsurname.
Note
Classattributesaresharedamongstallinstances,whileinstanceattributesarenot;
therefore,youshoulduseclassattributestoprovidethestatesandbehaviorstobeshared
byallinstances,anduseinstanceattributesfordatathatbelongsjusttoonespecificobject.
WOW! eBook
www.wowebook.org
Attributeshadowing
Whenyousearchforanattributeinanobject,ifitisnotfound,Pythonkeepssearchingin
theclassthatwasusedtocreatethatobject(andkeepssearchinguntilit’seitherfoundor
theendoftheinheritancechainisreached).Thisleadstoaninterestingshadowing
behavior.Let’slookatanexample:
oop/class.attribute.shadowing.py
classPoint():
x=10
y=7
p=Point()
print(p.x)#10(fromclassattribute)
print(p.y)#7(fromclassattribute)
p.x=12#pgetsitsown'x'attribute
print(p.x)#12(nowfoundontheinstance)
print(Point.x)#10(classattributestillthesame)
delp.x#wedeleteinstanceattribute
print(p.x)#10(nowsearchhastogoagaintofindclassattr)
p.z=3#let'smakeita3Dpoint
print(p.z)#3
print(Point.z)
#AttributeError:typeobject'Point'hasnoattribute'z'
Theprecedingcodeisveryinteresting.WehavedefinedaclasscalledPointwithtwo
classattributes,xandy.Whenwecreateaninstance,p,youcanseethatwecanprintboth
xandyfromp‘snamespace(p.xandp.y).WhathappenswhenwedothatisthatPython
doesn’tfindanyxoryattributesontheinstance,andthereforesearchestheclass,and
findsthemthere.
Thenwegivepitsownxattributebyassigningp.x=12.Thisbehaviormayappearabit
weirdatfirst,butifyouthinkaboutit,it’sexactlythesameaswhathappensinafunction
thatdeclaresx=12whenthereisaglobalx=10outside.Weknowthatx=12won’t
affecttheglobalone,andforclassesandinstances,itisexactlythesame.
Afterassigningp.x=12,whenweprintit,thesearchdoesn’tneedtoreadtheclass
attributes,becausexisfoundontheinstance,thereforeweget12printedout.
WealsoprintPoint.xwhichreferstoxintheclassnamespace.
Andthen,wedeletexfromthenamespaceofp,whichmeansthat,onthenextline,when
weprintitagain,Pythonwillgoagainandsearchforitintheclass,becauseitwon’tbe
foundintheinstanceanymore.
Thelastthreelinesshowyouthatassigningattributestoaninstancedoesn’tmeanthat
theywillbefoundintheclass.Instancesgetwhateverisintheclass,buttheoppositeis
nottrue.
WOW! eBook
www.wowebook.org
Whatdoyouthinkaboutputtingthexandycoordinatesasclassattributes?Doyouthink
itwasagoodidea?
WOW! eBook
www.wowebook.org
I,me,andmyself–usingtheselfvariable
Fromwithinaclassmethodwecanrefertoaninstancebymeansofaspecialargument,
calledselfbyconvention.selfisalwaysthefirstattributeofaninstancemethod.Let’s
examinethisbehaviortogetherwithhowwecanshare,notjustattributes,butmethods
withallinstances.
oop/class.self.py
classSquare():
side=8
defarea(self):#selfisareferencetoaninstance
returnself.side**2
sq=Square()
print(sq.area())#64(sideisfoundontheclass)
print(Square.area(sq))#64(equivalenttosq.area())
sq.side=10
print(sq.area())#100(sideisfoundontheinstance)
Notehowtheareamethodisusedbysq.Thetwocalls,Square.area(sq)and
sq.area(),areequivalent,andteachushowthemechanismworks.Eitheryoupassthe
instancetothemethodcall(Square.area(sq)),whichwithinthemethodwillbecalled
self,oryoucanuseamorecomfortablesyntax:sq.area()andPythonwilltranslatethat
foryoubehindthecurtains.
Let’slookatabetterexample:
oop/class.price.py
classPrice():
deffinal_price(self,vat,discount=0):
"""Returnspriceafterapplyingvatandfixeddiscount."""
return(self.net_price*(100+vat)/100)-discount
p1=Price()
p1.net_price=100
print(Price.final_price(p1,20,10))#110(100*1.2-10)
print(p1.final_price(20,10))#equivalent
Theprecedingcodeshowsyouthatnothingpreventsusfromusingargumentswhen
declaringmethods.Wecanusetheexactsamesyntaxasweusedwiththefunction,butwe
needtorememberthatthefirstargumentwillalwaysbetheinstance.
WOW! eBook
www.wowebook.org
Initializinganinstance
Haveyounoticedhow,beforecallingp1.final_price(...),wehadtoassignnet_price
top1?Thereisabetterwaytodoit.Inotherlanguages,thiswouldbecalleda
constructor,butinPython,it’snot.Itisactuallyaninitializer,sinceitworksonan
alreadycreatedinstance,andthereforeit’scalled__init__.It’samagicmethod,whichis
runrightaftertheobjectiscreated.Pythonobjectsalsohavea__new__method,whichis
theactualconstructor.Inpractice,it’snotsocommontohavetooverrideitthough,it’sa
practicethatismostlyusedwhencodingmetaclasses,whichisafairlyadvancedtopicthat
wewon’texploreinthebook.
oop/class.init.py
classRectangle():
def__init__(self,sideA,sideB):
self.sideA=sideA
self.sideB=sideB
defarea(self):
returnself.sideA*self.sideB
r1=Rectangle(10,4)
print(r1.sideA,r1.sideB)#104
print(r1.area())#40
r2=Rectangle(7,3)
print(r2.area())#21
Thingsarefinallystartingtotakeshape.Whenanobjectiscreated,the__init__method
isautomaticallyrunforus.Inthiscase,Icodeditsothatwhenwecreateanobject(by
callingtheclassnamelikeafunction),wepassargumentstothecreationcall,likewe
wouldonanyregularfunctioncall.Thewaywepassparametersfollowsthesignatureof
the__init__method,andtherefore,inthetwocreationstatements,10and7willbesideA
forr1andr2respectively,while4and3willbesideB.Youcanseethatthecalltoarea()
fromr1andr2reflectsthattheyhavedifferentinstancearguments.
Settingupobjectsinthiswayismuchnicerandconvenient.
WOW! eBook
www.wowebook.org
OOPisaboutcodereuse
Bynowitshouldbeprettyclear:OOPisallaboutcodereuse.Wedefineaclass,we
createinstances,andthoseinstancesusemethodsthataredefinedonlyintheclass.They
willbehavedifferentlyaccordingtohowtheinstanceshavebeensetupbytheinitializer.
Inheritanceandcomposition
Butthisisjusthalfofthestory,OOPismuchmorepowerful.Wehavetwomaindesign
constructstoexploit:inheritanceandcomposition.
InheritancemeansthattwoobjectsarerelatedbymeansofanIs-Atypeofrelationship.
Ontheotherhand,compositionmeansthattwoobjectsarerelatedbymeansofaHas-A
typeofrelationship.It’sallveryeasytoexplainwithanexample:
oop/class.inheritance.py
classEngine():
defstart(self):
pass
defstop(self):
pass
classElectricEngine(Engine):#Is-AEngine
pass
classV8Engine(Engine):#Is-AEngine
pass
classCar():
engine_cls=Engine
def__init__(self):
self.engine=self.engine_cls()#Has-AEngine
defstart(self):
print(
'Startingengine{0}forcar{1}...Wroom,wroom!'
.format(
self.engine.__class__.__name__,
self.__class__.__name__)
)
self.engine.start()
defstop(self):
self.engine.stop()
classRaceCar(Car):#Is-ACar
engine_cls=V8Engine
classCityCar(Car):#Is-ACar
engine_cls=ElectricEngine
classF1Car(RaceCar):#Is-ARaceCarandalsoIs-ACar
WOW! eBook
www.wowebook.org
engine_cls=V8Engine
car=Car()
racecar=RaceCar()
citycar=CityCar()
f1car=F1Car()
cars=[car,racecar,citycar,f1car]
forcarincars:
car.start()
"""Prints:
StartingengineEngineforcarCar...Wroom,wroom!
StartingengineV8EngineforcarRaceCar...Wroom,wroom!
StartingengineElectricEngineforcarCityCar...Wroom,wroom!
StartingengineV8EngineforcarF1Car...Wroom,wroom!
"""
TheprecedingexampleshowsyouboththeIs-AandHas-Atypesofrelationshipsbetween
objects.Firstofall,let’sconsiderEngine.It’sasimpleclassthathastwomethods,start
andstop.WethendefineElectricEngineandV8Engine,whichbothinheritfromEngine.
Youcanseethatbythefactthatwhenwedefinethem,weputEnginewithinthebraces
aftertheclassname.
ThismeansthatbothElectricEngineandV8Engineinheritattributesandmethodsfrom
theEngineclass,whichissaidtobetheirbaseclass.
Thesamehappenswithcars.CarisabaseclassforbothRaceCarandCityCar.RaceCaris
alsothebaseclassforF1Car.AnotherwayofsayingthisisthatF1Carinheritsfrom
RaceCar,whichinheritsfromCar.Therefore,F1CarIs-ARaceCarandRaceCarIs-ACar.
Becauseofthetransitiveproperty,wecansaythatF1CarIs-ACaraswell.CityCartoo,IsACar.
WhenwedefineclassA(B):pass,wesayAisthechildofB,andBistheparentofA.
parentandbasearesynonyms,aswellaschildandderived.Also,wesaythataclass
inheritsfromanotherclass,orthatitextendsit.
Thisistheinheritancemechanism.
Ontheotherhand,let’sgobacktothecode.Eachclasshasaclassattribute,engine_cls,
whichisareferencetotheengineclasswewanttoassigntoeachtypeofcar.Carhasa
genericEngine,whilethetworacecarshaveapowerfulV8engine,andthecitycarhasan
electricone.
Whenacariscreatedintheinitializermethod__init__,wecreateaninstanceof
whateverengineclassisassignedtothecar,andsetitasengineinstanceattribute.
Itmakessensetohaveengine_clssharedamongstallclassinstancesbecauseit’squite
likelythatthesameinstancesofacarwillhavethesamekindofengine.Ontheother
hand,itwouldn’tbegoodtohaveonesingleengine(aninstanceofanyEngineclass)asa
classattribute,becausewewouldbesharingoneengineamongstallinstances,whichis
incorrect.
WOW! eBook
www.wowebook.org
ThetypeofrelationshipbetweenacaranditsengineisaHas-Atype.AcarHas-Aengine.
Thisiscalledcomposition,andreflectsthefactthatobjectscanbemadeofmanyother
objects.AcarHas-Aengine,gears,wheels,aframe,doors,seats,andsoon.
WhendesigningOOPcode,itisofvitalimportancetodescribeobjectsinthiswaysothat
wecanuseinheritanceandcompositioncorrectlytostructureourcodeinthebestway.
Beforeweleavethisparagraph,let’scheckifItoldyouthetruthwithanotherexample:
oop/class.issubclass.isinstance.py
car=Car()
racecar=RaceCar()
f1car=F1Car()
cars=[(car,'car'),(racecar,'racecar'),(f1car,'f1car')]
car_classes=[Car,RaceCar,F1Car]
forcar,car_nameincars:
forclass_incar_classes:
belongs=isinstance(car,class_)
msg='isa'ifbelongselse'isnota'
print(car_name,msg,class_.__name__)
"""Prints:
carisaCar
carisnotaRaceCar
carisnotaF1Car
racecarisaCar
racecarisaRaceCar
racecarisnotaF1Car
f1carisaCar
f1carisaRaceCar
f1carisaF1Car
"""
Asyoucansee,carisjustaninstanceofCar,whileracecarisaninstanceofRaceCar
(andofCarbyextension)andf1carisaninstanceofF1Car(andofbothRaceCarandCar,
byextension).AbananaisaninstanceofBanana.But,also,itisaFruit.Also,itisFood,
right?Thisisthesameconcept.
Tocheckifanobjectisaninstanceofaclass,usetheisinstancemethod.Itis
recommendedoversheertypecomparison(type(object)==Class).
Let’salsocheckinheritance,samesetup,withdifferentforloops:
oop/class.issubclass.isinstance.py
forclass1incar_classes:
forclass2incar_classes:
is_subclass=issubclass(class1,class2)
msg='{0}asubclassof'.format(
'is'ifis_subclasselse'isnot')
print(class1.__name__,msg,class2.__name__)
"""Prints:
CarisasubclassofCar
WOW! eBook
www.wowebook.org
CarisnotasubclassofRaceCar
CarisnotasubclassofF1Car
RaceCarisasubclassofCar
RaceCarisasubclassofRaceCar
RaceCarisnotasubclassofF1Car
F1CarisasubclassofCar
F1CarisasubclassofRaceCar
F1CarisasubclassofF1Car
"""
Interestingly,welearnthataclassisasubclassofitself.Checktheoutputofthepreceding
exampletoseethatitmatchestheexplanationIprovided.
Note
Onethingtonoticeaboutconventionsisthatclassnamesarealwayswrittenusing
CapWords,whichmeansThisWayIsCorrect,asopposedtofunctionsandmethods,which
arewrittenthis_way_is_correct.Also,wheninthecodeyouwanttouseanamewhichisa
Python-reservedkeywordorbuilt-infunctionorclass,theconventionistoaddatrailing
underscoretothename.Inthefirstforloopexample,I’mloopingthroughtheclass
namesusingforclass_in…,becauseclassisareservedword.Butyoualreadyknewall
thisbecauseyouhavethoroughlystudiedPEP8,right?
TohelpyoupicturethedifferencebetweenIs-AandHas-A,takealookatthefollowing
diagram:
WOW! eBook
www.wowebook.org
Accessingabaseclass
We’vealreadyseenclassdeclarationslikeclassClassA:passandclass
ClassB(BaseClassName):pass.Whenwedon’tspecifyabaseclassexplicitly,Python
willsetthespecialobjectclassasthebaseclassfortheonewe’redefining.Ultimately,all
classesderivefromobject.Notethat,ifyoudon’tspecifyabaseclass,bracesare
optional.
Therefore,writingclassA:passorclassA():passorclassA(object):passis
exactlythesamething.objectisaspecialclassinthatithasthemethodsthatarecommon
toallPythonclasses,anditdoesn’tallowyoutosetanyattributesonit.
Let’sseehowwecanaccessabaseclassfromwithinaclass.
oop/super.duplication.py
classBook:
def__init__(self,title,publisher,pages):
self.title=title
self.publisher=publisher
self.pages=pages
classEbook(Book):
def__init__(self,title,publisher,pages,format_):
self.title=title
self.publisher=publisher
self.pages=pages
self.format_=format_
Takealookattheprecedingcode.IhighlightedthepartofEbookinitializationthatis
duplicatedfromitsbaseclassBook.Thisisquitebadpracticebecausewenowhavetwo
setsofinstructionsthataredoingthesamething.Moreover,anychangeinthesignatureof
Book.__init__willnotreflectinEbook.WeknowthatEbookIs-ABook,andthereforewe
wouldprobablywantchangestobereflectedinthechildrenclasses.
Let’sseeonewaytofixthisissue:
oop/super.explicit.py
classBook:
def__init__(self,title,publisher,pages):
self.title=title
self.publisher=publisher
self.pages=pages
classEbook(Book):
def__init__(self,title,publisher,pages,format_):
Book.__init__(self,title,publisher,pages)
self.format_=format_
ebook=Ebook('LearningPython','PacktPublishing',360,'PDF')
print(ebook.title)#LearningPython
print(ebook.publisher)#PacktPublishing
print(ebook.pages)#360
print(ebook.format_)#PDF
WOW! eBook
www.wowebook.org
Now,that’sbetter.Wehaveremovedthatnastyduplication.Basically,wetellPythonto
callthe__init__methodoftheBookclass,andwefeedselftothecall,makingsurethat
webindthatcalltothepresentinstance.
Ifwemodifythelogicwithinthe__init__methodofBook,wedon’tneedtotouchEbook,
itwillautoadapttothechange.
Thisapproachisgood,butwecanstilldoabitbetter.SaythatwechangeBook‘snameto
Liber,becausewe’vefalleninlovewithLatin.Wehavetochangethe__init__method
ofEbooktoreflectthechange.Thiscanbeavoidedbyusingsuper.
oop/super.implicit.py
classBook:
def__init__(self,title,publisher,pages):
self.title=title
self.publisher=publisher
self.pages=pages
classEbook(Book):
def__init__(self,title,publisher,pages,format_):
super().__init__(title,publisher,pages)
#Anotherwaytodothesamethingis:
#super(Ebook,self).__init__(title,publisher,pages)
self.format_=format_
ebook=Ebook('LearningPython','PacktPublishing',360,'PDF')
print(ebook.title)#LearningPython
print(ebook.publisher)#PacktPublishing
print(ebook.pages)#360
print(ebook.format_)#PDF
superisafunctionthatreturnsaproxyobjectthatdelegatesmethodcallstoaparentor
siblingclass.Inthiscase,itwilldelegatethatcallto__init__totheBookclass,andthe
beautyofthismethodisthatnowwe’reevenfreetochangeBooktoLiberwithouthaving
totouchthelogicinthe__init__methodofEbook.
Nowthatweknowhowtoaccessabaseclassfromachild,let’sexplorePython’smultiple
inheritance.
WOW! eBook
www.wowebook.org
Multipleinheritance
Apartfromcomposingaclassusingmorethanonebaseclass,whatisofinteresthereis
howanattributesearchisperformed.Takealookatthefollowingdiagram:
WOW! eBook
www.wowebook.org
Asyoucansee,ShapeandPlotteractasbaseclassesforalltheothers.Polygoninherits
directlyfromthem,RegularPolygoninheritsfromPolygon,andbothRegularHexagon
andSquareinheritfromRegulaPolygon.NotealsothatShapeandPlotterimplicitly
inheritfromobject,thereforewehavewhatiscalledadiamondor,insimplerterms,
morethanonepathtoreachabaseclass.We’llseewhythismattersinafewmoments.
Let’stranslateitintosomesimplecode:
oop/multiple.inheritance.py
classShape:
geometric_type='GenericShape'
defarea(self):#Thisactsasplaceholderfortheinterface
raiseNotImplementedError
defget_geometric_type(self):
returnself.geometric_type
classPlotter:
defplot(self,ratio,topleft):
#Imaginesomeniceplottinglogichere…
print('Plottingat{},ratio{}.'.format(
topleft,ratio))
classPolygon(Shape,Plotter):#baseclassforpolygons
geometric_type='Polygon'
classRegularPolygon(Polygon):#Is-APolygon
geometric_type='RegularPolygon'
def__init__(self,side):
self.side=side
classRegularHexagon(RegularPolygon):#Is-ARegularPolygon
geometric_type='RegularHexagon'
defarea(self):
return1.5*(3**.5*self.side**2)
classSquare(RegularPolygon):#Is-ARegularPolygon
geometric_type='Square'
defarea(self):
returnself.side*self.side
hexagon=RegularHexagon(10)
print(hexagon.area())#259.8076211353316
print(hexagon.get_geometric_type())#RegularHexagon
hexagon.plot(0.8,(75,77))#Plottingat(75,77),ratio0.8.
square=Square(12)
print(square.area())#144
print(square.get_geometric_type())#Square
square.plot(0.93,(74,75))#Plottingat(74,75),ratio0.93.
WOW! eBook
www.wowebook.org
Takealookattheprecedingcode:theclassShapehasoneattribute,geometric_type,and
twomethods:areaandget_geometric_type.It’squitecommontousebaseclasses(like
Shape,inourexample)todefineaninterface:methodsforwhichchildrenmustprovidean
implementation.Therearedifferentandbetterwaystodothis,butIwanttokeepthis
exampleassimpleaspossible.
WealsohavethePlotterclass,whichaddstheplotmethod,therebyprovidingplotting
capabilitiesforanyclassthatinheritsfromit.Ofcourse,theplotimplementationisjusta
dummyprintinthisexample.ThefirstinterestingclassisPolygon,whichinheritsfrom
bothShapeandPlotter.
Therearemanytypesofpolygons,oneofwhichistheregularone,whichisboth
equiangular(allanglesareequal)andequilateral(allsidesareequal),sowecreatethe
RegularPolygonclassthatinheritsfromPolygon.Becauseforaregularpolygon,allsides
areequal,wecanimplementasimple__init__methodonRegularPolygon,whichtakes
thelengthoftheside.Finally,wecreatetheRegularHexagonandSquareclasses,which
bothinheritfromRegularPolygon.
Thisstructureisquitelong,buthopefullygivesyouanideaofhowtospecializethe
classificationofyourobjectswhenyoudesignthecode.
Now,pleasetakealookatthelasteightlines.NotethatwhenIcalltheareamethodon
hexagonandsquare,Igetthecorrectareaforboth.Thisisbecausetheybothprovidethe
correctimplementationlogicforit.Also,Icancallget_geometric_typeonbothofthem,
eventhoughitisnotdefinedontheirclasses,andPythonhastogoallthewayuptoShape
tofindanimplementationforit.Notethat,eventhoughtheimplementationisprovidedin
theShapeclass,theself.geometric_typeusedforthereturnvalueiscorrectlytaken
fromthecallerinstance.
Theplotmethodcallsarealsointeresting,andshowyouhowyoucanenrichyourobjects
withcapabilitiestheywouldn’totherwisehave.Thistechniqueisverypopularinweb
frameworkssuchasDjango(whichwe’llexploreintwolaterchapters),whichprovides
specialclassescalledmixins,whosecapabilitiesyoucanjustuseoutofthebox.Allyou
havetodoistodefinethedesiredmixinasonethebaseclassesforyourown,andthat’sit.
Multipleinheritanceispowerful,butcanalsogetreallymessy,soweneedtomakesure
weunderstandwhathappenswhenweuseit.
Methodresolutionorder
Bynow,weknowthatwhenyouaskforsomeobject.attribute,andattributeisnot
foundonthatobject,Pythonstartssearchingintheclasssomeobjectwascreatedfrom.If
it’snotthereeither,Pythonsearchesuptheinheritancechainuntileitherattributeis
foundortheobjectclassisreached.Thisisquitesimpletounderstandiftheinheritance
chainisonlycomprisedofsingleinheritancesteps,whichmeansthatclasseshaveonly
oneparent.However,whenmultipleinheritanceisinvolved,therearecaseswhenit’snot
straightforwardtopredictwhatwillbethenextclassthatwillbesearchedforifan
attributeisnotfound.
WOW! eBook
www.wowebook.org
Pythonprovidesawaytoalwaysknowwhatistheorderinwhichclassesaresearchedon
attributelookup:themethodresolutionorder.
Note
Themethodresolutionorder(MRO)istheorderinwhichbaseclassesaresearchedfor
amemberduringlookup.Fromversion2.3PythonusesanalgorithmcalledC3,which
guaranteesmonotonicity.
InPython2.2,new-styleclasseswereintroduced.Thewayyouwriteanew-styleclassin
Python2.*istodefineitwithanexplicitobjectbaseclass.Classicclasseswerenot
explicitlyinheritingfromobjectandhavebeenremovedinPython3.
Oneofthedifferencesbetweenclassicandnewstyle-classesinPython2.*isthatnewstyleclassesaresearchedwiththenewMRO.
Withregardstothepreviousexample,let’sseewhatistheMROfortheSquareclass:
oop/multiple.inheritance.py
print(square.__class__.__mro__)
#prints:
#(<class'__main__.Square'>,<class'__main__.RegularPolygon'>,
#<class'__main__.Polygon'>,<class'__main__.Shape'>,
#<class'__main__.Plotter'>,<class'object'>)
TogettotheMROofaclass,wecangofromtheinstancetoits__class__attributeand
fromthattoits__mro__attribute.Alternatively,wecouldhavecalledSquare.__mro__,or
Square.mro()directly,butifyouhavetodoitdynamically,it’smorelikelyyouwillhave
anobjectinyourhandsratherthanaclass.
NotethattheonlypointofdoubtisthebisectionafterPolygon,wheretheinheritance
chainbreaksintotwoways,oneleadstoShapeandtheothertoPlotter.Weknowby
scanningtheMROfortheSquareclassthatShapeissearchedbeforePlotter.
Whyisthisimportant?Well,imaginethefollowingcode:
oop/mro.simple.py
classA:
label='a'
classB(A):
label='b'
classC(A):
label='c'
classD(B,C):
pass
d=D()
print(d.label)#Hypotheticallythiscouldbeeither'b'or'c'
BothBandCinheritfromA,andDinheritsfrombothBandC.Thismeansthatthelookup
WOW! eBook
www.wowebook.org
forthelabelattributecanreachthetop(A)throughbothBorC.Accordingtowhichis
reachedfirst,wegetadifferentresult.
So,intheprecedingexampleweget'b',whichiswhatwewereexpecting,sinceBisthe
leftmostoneamongstbaseclassesofD.ButwhathappensifIremovethelabelattribute
fromB?Thiswouldbetheconfusingsituation:WillthealgorithmgoallthewayuptoAor
willitgettoCfirst?Let’sfindout!
oop/mro.py
classA:
label='a'
classB(A):
pass#was:label='b'
classC(A):
label='c'
classD(B,C):
pass
d=D()
print(d.label)#'c'
print(d.__class__.mro())#noticeanotherwaytogettheMRO
#prints:
#[<class'__main__.D'>,<class'__main__.B'>,
#<class'__main__.C'>,<class'__main__.A'>,<class'object'>]
So,welearnthattheMROisD-B-C-A-(object),whichmeanswhenweaskford.label,
weget'c',whichiscorrect.
Indaytodayprogramming,itisnotquitecommontohavetodealwiththeMRO,butthe
firsttimeyoufightagainstsomemixinfromaframework,Ipromiseyou’llbegladIspent
aparagraphexplainingit.
WOW! eBook
www.wowebook.org
Staticandclassmethods
Untilnow,wehavecodedclasseswithattributesintheformofdataandinstancemethods,
buttherearetwoothertypesofmethodsthatwecanplaceinsideaclass:staticmethods
andclassmethods.
Staticmethods
Asyoumayrecall,whenyoucreateaclassobject,Pythonassignsanametoit.Thatname
actsasanamespace,andsometimesitmakessensetogroupfunctionalitiesunderit.Static
methodsareperfectforthisusecasesinceunlikeinstancemethods,theyarenotpassed
anyspecialargument.Let’slookatanexampleofanimaginaryStringclass.
oop/static.methods.py
classString:
@staticmethod
defis_palindrome(s,case_insensitive=True):
#weallowonlylettersandnumbers
s=''.join(cforcinsifc.isalnum())#Studythis!
#Forcaseinsensitivecomparison,welower-cases
ifcase_insensitive:
s=s.lower()
forcinrange(len(s)//2):
ifs[c]!=s[-c-1]:
returnFalse
returnTrue
@staticmethod
defget_unique_words(sentence):
returnset(sentence.split())
print(String.is_palindrome(
'Radar',case_insensitive=False))#False:CaseSensitive
print(String.is_palindrome('Anutforajaroftuna'))#True
print(String.is_palindrome('NeverOdd,OrEven!'))#True
print(String.is_palindrome(
'InGirumImusNocteEtConsumimurIgni')#Latin!Show-off!
)#True
print(String.get_unique_words(
'Ilovepalindromes.Ireallyreallylovethem!'))
#{'them!','really','palindromes.','I','love'}
Theprecedingcodeisquiteinteresting.Firstofall,welearnthatstaticmethodsare
createdbysimplyapplyingthestaticmethoddecoratortothem.Youcanseethatthey
aren’tpassedanyspecialargumentso,apartfromthedecoration,theyreallyjustlooklike
functions.
Wehaveaclass,String,whichactsasacontainerforfunctions.Anotherapproachwould
betohaveaseparatemodulewithfunctionsinside.It’sreallyamatterofpreferencemost
ofthetime.
WOW! eBook
www.wowebook.org
Thelogicinsideis_palindromeshouldbestraightforwardforyoutounderstandbynow,
but,justincase,let’sgothroughit.Firstweremoveallcharactersfromsthatarenot
eitherlettersornumbers.Inordertodothis,weusethejoinmethodofastringobject(an
emptystringobject,inthiscase).Bycallingjoinonanemptystring,theresultisthatall
elementsintheiterableyoupasstojoinwillbeconcatenatedtogether.Wefeedjoina
generatorexpressionthatsays,takeanycharacterfromsifthecharacteriseither
alphanumericoranumber.Ihopeyouhavebeenabletofindthatoutbyyourself,maybe
usingtheinside-outtechniqueIshowedyouinoneoftheprecedingchapters.
Wethenlowercasesifcase_insensitiveisTrue,andthenweproceedtocheckifitisa
palindrome.Inordertodothis,wecomparethefirstandlastcharacters,thenthesecond
andthesecondtolast,andsoon.Ifatanypointwefindadifference,itmeansthestring
isn’tapalindromeandthereforewecanreturnFalse.Ontheotherhand,ifweexitthefor
loopnormally,itmeansnodifferenceswerefound,andwecanthereforesaythestringisa
palindrome.
Noticethatthiscodeworkscorrectlyregardlessofthelengthofthestring,thatis,ifthe
lengthisoddoreven.len(s)//2reacheshalfofs,andifsisanoddamountof
characterslong,themiddleonewon’tbechecked(likeinRaDaR,Disnotchecked),but
wedon’tcare;itwouldbecomparedwithitselfsoit’salwayspassingthatcheck.
get_unique_wordsismuchsimpler,itjustreturnsasettowhichwefeedalistwiththe
wordsfromasentence.Thesetclassremovesanyduplicationforus,thereforewedon’t
needtodoanythingelse.
TheStringclassprovidesusanicecontainernamespaceformethodsthataremeantto
workonstrings.IcouldhavecodedasimilarexamplewithaMathclass,andsomestatic
methodstoworkonnumbers,butIwantedtoshowyousomethingdifferent.
Classmethods
Classmethodsareslightlydifferentfrominstancemethodsinthattheyalsotakeaspecial
firstargument,butinthiscase,itistheclassobjectitself.Twoverycommonusecasesfor
codingclassmethodsaretoprovidefactorycapabilitytoaclassandtoallowbreakingup
staticmethods(whichyouhavetothencallusingtheclassname)withouthavingto
hardcodetheclassnameinyourlogic.Let’slookatanexampleofbothofthem.
oop/class.methods.factory.py
classPoint:
def__init__(self,x,y):
self.x=x
self.y=y
@classmethod
deffrom_tuple(cls,coords):#clsisPoint
returncls(*coords)
@classmethod
deffrom_point(cls,point):#clsisPoint
returncls(point.x,point.y)
WOW! eBook
www.wowebook.org
p=Point.from_tuple((3,7))
print(p.x,p.y)#37
q=Point.from_point(p)
print(q.x,q.y)#37
Intheprecedingcode,Ishowedyouhowtouseaclassmethodtocreateafactoryforthe
class.Inthiscase,wewanttocreateaPointinstancebypassingbothcoordinates(regular
creationp=Point(3,7)),butwealsowanttobeabletocreateaninstancebypassinga
tuple(Point.from_tuple)oranotherinstance(Point.from_point).
Withinthetwoclassmethods,theclsargumentreferstothePointclass.Aswithinstance
method,whichtakeselfasthefirstargument,classmethodtakeaclsargument.Both
selfandclsarenamedafteraconventionthatyouarenotforcedtofollowbutare
stronglyencouragedtorespect.ThisissomethingthatnoPythoncoderwouldchange
becauseitissostrongaconventionthatparsers,linters,andanytoolthatautomatically
doessomethingwithyourcodewouldexpect,soit’smuchbettertosticktoit.
Let’slookatanexampleoftheotherusecase:splittingastaticmethod.
oop/class.methods.split.py
classString:
@classmethod
defis_palindrome(cls,s,case_insensitive=True):
s=cls._strip_string(s)
#Forcaseinsensitivecomparison,welower-cases
ifcase_insensitive:
s=s.lower()
returncls._is_palindrome(s)
@staticmethod
def_strip_string(s):
return''.join(cforcinsifc.isalnum())
@staticmethod
def_is_palindrome(s):
forcinrange(len(s)//2):
ifs[c]!=s[-c-1]:
returnFalse
returnTrue
@staticmethod
defget_unique_words(sentence):
returnset(sentence.split())
print(String.is_palindrome('Anutforajaroftuna'))#True
print(String.is_palindrome('Anutforajarofbeans'))#False
Comparethiscodewiththepreviousversion.Firstofallnotethateventhough
is_palindromeisnowaclassmethod,wecallitinthesamewaywewerecallingitwhen
itwasastaticone.Thereasonwhywechangedittoaclassmethodisthatafterfactoring
outacoupleofpiecesoflogic(_strip_stringand_is_palindrome),weneedtogeta
WOW! eBook
www.wowebook.org
referencetothemandifwehavenoclsinourmethod,theonlyoptionwouldbetocall
themlikethis:String._strip_string(...)andString._is_palindrome(...),whichis
notgoodpractice,becausewewouldhardcodetheclassnameintheis_palindrome
method,therebyputtingourselvesintheconditionofhavingtomodifyitwheneverwe
wouldchangetheclassname.Usingclswillactastheclassname,whichmeansourcode
won’tneedanyamendments.
Notealsothat,bynamingthefactored-outmethodswithaleadingunderscore,Iam
hintingthatthosemethodsarenotsupposedtobecalledfromoutsidetheclass,butthis
willbethesubjectofthenextparagraph.
WOW! eBook
www.wowebook.org
Privatemethodsandnamemangling
IfyouhaveanybackgroundwithlanguageslikeJava,C#,C++,orsimilar,thenyouknow
theyallowtheprogrammertoassignaprivacystatustoattributes(bothdataandmethods).
Eachlanguagehasitsownslightlydifferentflavorforthis,butthegististhatpublic
attributesareaccessiblefromanypointinthecode,whileprivateonesareaccessibleonly
withinthescopetheyaredefinedin.
InPython,thereisnosuchthing.Everythingispublic;therefore,werelyonconventions
andonamechanismcallednamemangling.
Theconventionisasfollows:ifanattribute’snamehasnoleadingunderscoresitis
consideredpublic.Thismeansyoucanaccessitandmodifyitfreely.Whenthenamehas
oneleadingunderscore,theattributeisconsideredprivate,whichmeansit’sprobably
meanttobeusedinternallyandyoushouldnotuseitormodifyitfromtheoutside.Avery
commonusecaseforprivateattributesarehelpermethodsthataresupposedtobeusedby
publicones(possiblyincallchainsinconjunctionwithothermethods),andinternaldata,
likescalingfactors,oranyotherdatathatideallywewouldputinaconstant(avariable
thatcannotchange,but,surprise,surprise,Pythondoesn’thavethoseeither).
Thischaracteristicusuallyscarespeoplefromotherbackgroundsoff;theyfeelthreatened
bythelackofprivacy.Tobehonest,inmywholeprofessionalexperiencewithPython,
I’veneverheardanyonescreamingOhmyGod,wehaveaterriblebugbecausePython
lacksprivateattributes!Notonce,Iswear.
Thatsaid,thecallforprivacyactuallymakessensebecausewithoutit,yourisk
introducingbugsintoyourcodeforreal.Let’slookatasimpleexample:
oop/private.attrs.py
classA:
def__init__(self,factor):
self._factor=factor
defop1(self):
print('Op1withfactor{}...'.format(self._factor))
classB(A):
defop2(self,factor):
self._factor=factor
print('Op2withfactor{}...'.format(self._factor))
obj=B(100)
obj.op1()#Op1withfactor100…
obj.op2(42)#Op2withfactor42…
obj.op1()#Op1withfactor42…<-ThisisBAD
Intheprecedingcode,wehaveanattributecalled_factor,andlet’spretendit’svery
importantthatitisn’tmodifiedatruntimeaftertheinstanceiscreated,becauseop1
dependsonittofunctioncorrectly.We’venameditwithaleadingunderscore,butthe
issuehereisthatwhenwecallobj.op2(42),wemodifyit,andthisreflectsinsubsequent
WOW! eBook
www.wowebook.org
callstoop1.
Let’sfixthisundesiredbehaviorbyaddinganotherleadingunderscore:
oop/private.attrs.fixed.py
classA:
def__init__(self,factor):
self.__factor=factor
defop1(self):
print('Op1withfactor{}...'.format(self.__factor))
classB(A):
defop2(self,factor):
self.__factor=factor
print('Op2withfactor{}...'.format(self.__factor))
obj=B(100)
obj.op1()#Op1withfactor100…
obj.op2(42)#Op2withfactor42…
obj.op1()#Op1withfactor100…<-Wohoo!Nowit'sGOOD!
Wow,lookatthat!Nowit’sworkingasdesired.Pythoniskindofmagicandinthiscase,
whatishappeningisthatthenamemanglingmechanismhaskickedin.
Namemanglingmeansthatanyattributenamethathasatleasttwoleadingunderscores
andatmostonetrailingunderscore,like__my_attr,isreplacedwithanamethatincludes
anunderscoreandtheclassnamebeforetheactualname,like_ClassName__my_attr.
Thismeansthatwhenyouinheritfromaclass,themanglingmechanismgivesyour
privateattributetwodifferentnamesinthebaseandchildclassessothatnamecollisionis
avoided.Everyclassandinstanceobjectstoresreferencestotheirattributesinaspecial
attributecalled__dict__,solet’sinspectobj.__dict__toseenamemanglinginaction:
oop/private.attrs.py
print(obj.__dict__.keys())
#dict_keys(['_factor'])
Thisisthe_factorattributethatwefindintheproblematicversionofthisexample.But
lookattheonethatisusing__factor:
oop/private.attrs.fixed.py
print(obj.__dict__.keys())
#dict_keys(['_A__factor','_B__factor'])
See?objhastwoattributesnow,_A__factor(mangledwithintheAclass),and
_B__factor(mangledwithintheBclass).Thisisthemechanismthatmakespossiblethat
whenyoudoobj.__factor=42,__factorinAisn’tchanged,becauseyou’reactually
touching_B__factor,whichleaves_A__factorsafeandsound.
Ifyou’redesigningalibrarywithclassesthataremeanttobeusedandextendedbyother
developers,youwillneedtokeepthisinmindinordertoavoidunintentionaloverriding
WOW! eBook
www.wowebook.org
ofyourattributes.Bugslikethesecanbeprettysubtleandhardtospot.
WOW! eBook
www.wowebook.org
Thepropertydecorator
Anotherthingthatwouldbeacrimenottomentionisthepropertydecorator.Imagine
thatyouhaveanageattributeinaPersonclassandatsomepointyouwanttomakesure
thatwhenyouchangeitsvalue,you’realsocheckingthatageiswithinaproperrange,
like[18,99].Youcanwriteaccessormethods,likeget_age()andset_age()(alsocalled
gettersandsetters)andputthelogicthere.get_age()willmostlikelyjustreturnage,
whileset_age()willalsodotherangecheck.Theproblemisthatyoumayalreadyhavea
lotofcodeaccessingtheageattributedirectly,whichmeansyou’renowuptosomegood
(andpotentiallydangerousandtedious)refactoring.LanguageslikeJavaovercomethis
problembyusingtheaccessorpatternbasicallybydefault.ManyJavaIntegrated
DevelopmentEnvironments(IDEs)autocompleteanattributedeclarationbywriting
getterandsetteraccessormethodsstubsforyouonthefly.
Pythonissmarter,anddoesthiswiththepropertydecorator.Whenyoudecoratea
methodwithproperty,youcanusethenameofthemethodasifitwasadataattribute.
Becauseofthis,it’salwaysbesttorefrainfromputtinglogicthatwouldtakeawhileto
completeinsuchmethodsbecause,byaccessingthemasattributes,wearenotexpecting
towait.
Let’slookatanexample:
oop/property.py
classPerson:
def__init__(self,age):
self.age=age#anyonecanmodifythisfreely
classPersonWithAccessors:
def__init__(self,age):
self._age=age
defget_age(self):
returnself._age
defset_age(self):
if18<=age<=99:
self._age=age
else:
raiseValueError('Agemustbewithin[18,99]')
classPersonPythonic:
def__init__(self,age):
self._age=age
@property
defage(self):
returnself._age
@age.setter
defage(self,age):
if18<=age<=99:
WOW! eBook
www.wowebook.org
self._age=age
else:
raiseValueError('Agemustbewithin[18,99]')
person=PersonPythonic(39)
print(person.age)#39-Noticeweaccessasdataattribute
person.age=42#Noticeweaccessasdataattribute
print(person.age)#42
person.age=100#ValueError:Agemustbewithin[18,99]
ThePersonclassmaybethefirstversionwewrite.Thenwerealizeweneedtoputthe
rangelogicinplaceso,withanotherlanguage,wewouldhavetorewritePersonasthe
PersonWithAccessorsclass,andrefactorallthecodethatwasusingPerson.age.In
Python,werewritePersonasPersonPythonic(younormallywouldn’tchangethename,
ofcourse)sothattheageisstoredinaprivate_agevariable,andwedefineproperty
gettersandsettersusingthatdecoration,whichallowustokeepusingtheperson
instancesaswewerebefore.Agetterisamethodthatiscalledwhenweaccessan
attributeforreading.Ontheotherhand,asetterisamethodthatiscalledwhenweaccess
anattributetowriteit.Inotherlanguages,likeJavaforexample,it’scustomarytodefine
themasget_age()andset_age(intvalue),butIfindthePythonsyntaxmuchneater.It
allowsyoutostartwritingsimplecodeandrefactorlateron,onlywhenyouneedit,there
isnoneedtopolluteyourcodewithaccessorsonlybecausetheymaybehelpfulinthe
future.
Thepropertydecoratoralsoallowsforread-onlydata(nosetter)andforspecialactions
whentheattributeisdeleted.Pleaserefertotheofficialdocumentationtodigdeeper.
WOW! eBook
www.wowebook.org
Operatoroverloading
IfindPython’sapproachtooperatoroverloadingtobebrilliant.Tooverloadanoperator
meanstogiveitameaningaccordingtothecontextinwhichitisused.Forexample,the+
operatormeansadditionwhenwedealwithnumbers,butconcatenationwhenwedeal
withsequences.
InPython,whenyouuseoperators,you’remostlikelycallingthespecialmethodsofsome
objectsbehindthescenes.Forexample,thecalla[k]roughlytranslatesto
type(a).__getitem__(a,k).
Asanexample,let’screateaclassthatstoresastringandevaluatestoTrueif'42'ispart
ofthatstring,andFalseotherwise.Also,let’sgivetheclassalengthpropertywhich
correspondstothatofthestoredstring.
oop/operator.overloading.py
classWeird:
def__init__(self,s):
self._s=s
def__len__(self):
returnlen(self._s)
def__bool__(self):
return'42'inself._s
weird=Weird('Hello!Iam9yearsold!')
print(len(weird))#24
print(bool(weird))#False
weird2=Weird('Hello!Iam42yearsold!')
print(len(weird2))#25
print(bool(weird2))#True
Thatwasfun,wasn’tit?Forthecompletelistofmagicmethodsthatyoucanoverridein
ordertoprovideyourcustomimplementationofoperatorsforyourclasses,pleasereferto
thePythondatamodelintheofficialdocumentation.
WOW! eBook
www.wowebook.org
Polymorphism–abriefoverview
ThewordpolymorphismcomesfromtheGreekpolys(many,much)andmorphē(form,
shape),anditsmeaningistheprovisionofasingleinterfaceforentitiesofdifferenttypes.
Inourcarexample,wecallengine.start(),regardlessofwhatkindofengineitis.As
longasitexposesthestartmethod,wecancallit.That’spolymorphisminaction.
Inotherlanguages,likeJava,inordertogiveafunctiontheabilitytoacceptdifferent
typesandcallamethodonthem,thosetypesneedtobecodedinsuchawaythatthey
shareaninterface.Inthisway,thecompilerknowsthatthemethodwillbeavailable
regardlessofthetypeoftheobjectthefunctionisfed(aslongasitextendstheproper
interface,ofcourse).
InPython,thingsaredifferent.Polymorphismisimplicit,nothingpreventsyoutocalla
methodonanobject,therefore,technically,thereisnoneedtoimplementinterfacesor
otherpatterns.
Thereisaspecialkindofpolymorphismcalledadhocpolymorphism,whichiswhatwe
sawinthelastparagraph:operatoroverloading.Theabilityofanoperatortochange
shape,accordingtothetypeofdataitisfed.
Icannotspendtoomuchtimeonpolymorphism,butIencourageyoutocheckitoutby
yourself,itwillexpandyourunderstandingofOOP.Goodluck!
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Writingacustomiterator
Nowwehaveallthetoolstoappreciatehowwecanwriteourowncustomiterator.Let’s
firstdefinewhatisaniterableandaniterator:
Iterable:Anobjectissaidtobeiterableifit’scapableofreturningitsmembersone
atatime.Lists,tuples,strings,dicts,arealliterables.Customobjectsthatdefine
eitherof__iter__or__getitem__methodsarealsoiterables.
Iterator:Anobjectissaidtobeaniteratorifitrepresentsastreamofdata.Acustom
iteratorisrequiredtoprovideanimplementationfor__iter__thatreturnstheobject
itself,andanimplementationfor__next__,whichreturnsthenextitemofthedata
streamuntilthestreamisexhausted,atwhichpointallsuccessivecallsto__next__
simplyraisetheStopIterationexception.Built-infunctionssuchasiterandnext
aremappedtocall__iter__and__next__onanobject,behindthescenes.
Let’swriteaniteratorthatreturnsalltheoddcharactersfromastringfirst,andthenthe
evenones.
iterators/iterator.py
classOddEven:
def__init__(self,data):
self._data=data
self.indexes=(list(range(0,len(data),2))+
list(range(1,len(data),2)))
def__iter__(self):
returnself
def__next__(self):
ifself.indexes:
returnself._data[self.indexes.pop(0)]
raiseStopIteration
oddeven=OddEven('ThIsIsCoOl!')
print(''.join(cforcinoddeven))#TIICO!hssol
oddeven=OddEven('HoLa')#ormanually…
it=iter(oddeven)#thiscallsoddeven.__iter__internally
print(next(it))#H
print(next(it))#L
print(next(it))#o
print(next(it))#a
So,weneededtoprovideanimplementationfor__iter__whichreturnedtheobjectitself,
andthenonefor__next__.Let’sgothroughit.Whatneedstohappenisthatwereturn
_data[0],_data[2],_data[4],…,_data[1],_data[3],_data[5],…untilwehave
returnedeveryiteminthedata.Inordertodothis,wepreparealist,indexes,like[0,2,4,
6,…,1,3,5,…],andwhilethereisatleastanelementinit,wepopthefirstoneand
returntheelementfromthedatathatisatthatposition,therebyachievingourgoal.When
indexesisempty,weraiseStopIteration,asrequiredbytheiteratorprotocol.
WOW! eBook
www.wowebook.org
Thereareotherwaystoachievethesameresult,sogoaheadandtrytocodeadifferent
oneyourself.Makesuretheendresultworksforalledgecases,emptysequences,
sequencesoflength1,2,andsoon.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,wesawdecorators,discoveredthereasonsforhavingthem,andafew
examplesusingoneormoreatthesametime.Wealsosawdecoratorsthattakearguments,
whichareusuallyusedasdecoratorfactories.
Wescratchedthesurfaceofobject-orientedprogramminginPython.Wecoveredallthe
basicsinawaythatyoushouldnowbeabletounderstandfairlyeasilythecodethatwill
comeinfuturechapters.Wetalkedaboutallkindsofmethodsandattributesthatonecan
writeinaclass,weexploredinheritanceversuscomposition,methodoverriding,
properties,operatoroverloading,andpolymorphism.
Attheend,weverybrieflytouchedbaseoniterators,sonowyouhavealltheknowledge
toalsounderstandgeneratorsmoredeeply.
Inthenextchapter,wetakeasteepturn.Itwillstartthesecondhalfofthebook,whichis
muchmoreproject-orientedso,fromnowon,itwillbelesstheoryandmorecode,Ihope
youwillenjoyfollowingtheexamplesandgettingyourhandsdirty,verydirty.
Theysaythatasmoothseanevermadeaskillfulsailor,sokeepexploring,breakthings,
readtheerrormessagesaswellasthedocumentation,andlet’sseeifwecangettoseethat
whiterabbit.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter7.Testing,Profiling,andDealing
withExceptions
“Codewithouttestsisbrokenbydesign.” —JacobKaplan-Moss
JacobKaplan-MossisoneofthecoredevelopersoftheDjangowebframework.We’re
goingtoexploreitinthenextchapters.Istronglyagreewiththisquoteofhis.Ibelieve
codewithouttestsshouldn’tbedeployedtoproduction.
Whyaretestssoimportant?Well,forone,theygiveyoupredictability.Or,atleast,they
helpyouachievehighpredictability.Unfortunately,thereisalwayssomebugthatsneaks
intoourcode.Butwedefinitelywantourcodetobeaspredictableaspossible.Whatwe
don’twantistohaveasurprise,ourcodebehavinginanunpredictableway.Wouldyoube
happytoknowthatthesoftwarethatchecksonthesensorsoftheplanethatistakingyou
onholidayssometimesgoescrazy?No,probablynot.
Thereforeweneedtotestourcode,weneedtocheckthatitsbehavioriscorrect,thatit
worksasexpectedwhenitdealswithedgecases,thatitdoesn’thangwhenthe
componentsit’stalkingtoaredown,thattheperformancesarewellwithintheacceptable
range,andsoon.
Thischapterisallaboutthistopic,makingsurethatyourcodeispreparedtofacethe
scaryoutsideworld,thatisfastenoughandthatitcandealwithunexpectedorexceptional
conditions.
We’regoingtoexploretesting,includingabriefintroductiontotest-drivendevelopment
(TDD),whichisoneofmyfavoriteworkingmethodologies.Then,we’regoingtoexplore
theworldofexceptions,andfinallywe’regoingtotalkalittlebitaboutperformancesand
profiling.Deepbreath,andherewego…
WOW! eBook
www.wowebook.org
Testingyourapplication
Therearemanydifferentkindsoftests,somanyinfactthatcompaniesoftenhavea
dedicateddepartment,calledqualityassurance(QA),madeupofindividualsthatspend
theirdaytestingthesoftwarethecompanydevelopersproduce.
Tostartmakinganinitialclassification,wecandividetestsintotwobroadcategories:
white-boxandblack-boxtests.
White-boxtestsarethosewhichexercisetheinternalsofthecode,theyinspectitdownto
averyfinelevelofgranularity.Ontheotherhand,black-boxtestsarethosewhich
considerthesoftwareundertestingasifbeingwithinabox,theinternalsofwhichare
ignored.Eventhetechnology,orthelanguageusedinsidetheboxisnotimportantfor
black-boxtests.Whattheydoistopluginputtooneendoftheboxandverifytheoutput
attheotherend,andthat’sit.
Note
Thereisalsoanin-betweencategory,calledgray-boxtesting,thatinvolvestestinga
systeminthesamewaywedowiththeblack-boxapproach,buthavingsomeknowledge
aboutthealgorithmsanddatastructuresusedtowritethesoftwareandonlypartialaccess
toitssourcecode.
Therearemanydifferentkindsoftestsinthesecategories,eachofwhichservesadifferent
purpose.Justtogiveyouanidea,here’safew:
Front-endtestsmakesurethattheclientsideofyourapplicationisexposingthe
informationthatitshould,allthelinks,thebuttons,theadvertising,everythingthat
needstobeshowntotheclient.Itmayalsoverifythatitispossibletowalkacertain
paththroughtheuserinterface.
Scenariotestsmakeuseofstories(orscenarios)thathelpthetesterworkthrougha
complexproblemortestapartofthesystem.
Integrationtestsverifythebehaviorofthevariouscomponentsofyourapplication
whentheyareworkingtogethersendingmessagesthroughinterfaces.
Smoketestsareparticularlyusefulwhenyoudeployanewupdateonyour
application.Theycheckwhetherthemostessential,vitalpartsofyourapplicationare
stillworkingastheyshouldandthattheyarenotonfire.Thistermcomesfromwhen
engineerstestedcircuitsbymakingsurenothingwassmoking.
Acceptancetests,oruseracceptancetesting(UAT)iswhatadeveloperdoeswitha
productowner(forexample,inaSCRUMenvironment)todetermineiftheworkthat
wascommissionedwascarriedoutcorrectly.
Functionaltestsverifythefeaturesorfunctionalitiesofyoursoftware.
Destructiveteststakedownpartsofyoursystem,simulatingafailure,inorderto
establishhowwelltheremainingpartsofthesystemperform.Thesekindsoftestsare
performedextensivelybycompaniesthatneedtoprovideanextremelyreliable
service,suchasAmazon,forexample.
Performancetestsaimtoverifyhowwellthesystemperformsunderaspecificload
WOW! eBook
www.wowebook.org
ofdataortrafficsothat,forexample,engineerscangetabetterunderstandingof
whicharethebottlenecksinthesystemthatcouldbringitdowntoitskneesina
heavyloadsituation,orthosewhichpreventscalability.
Usabilitytests,andthecloselyrelateduserexperience(UX)tests,aimtocheckif
theuserinterfaceissimpleandeasytounderstandanduse.Theyaimtoprovideinput
tothedesignerssothattheuserexperienceisimproved.
Securityandpenetrationtestsaimtoverifyhowwellthesystemisprotected
againstattacksandintrusions.
Unittestshelpthedevelopertowritethecodeinarobustandconsistentway,
providingthefirstlineoffeedbackanddefenseagainstcodingmistakes,refactoring
mistakes,andsoon.
Regressiontestsprovidethedeveloperwithusefulinformationaboutafeaturebeing
compromisedinthesystemafteranupdate.Someofthecausesforasystembeing
saidtohavearegressionareanoldbugcomingbacktolife,anexistingfeaturebeing
compromised,oranewissuebeingintroduced.
Manybooksandarticleshavebeenwrittenabouttesting,andIhavetopointyoutothose
resourcesifyou’reinterestedinfindingoutmoreaboutallthedifferentkindsoftests.In
thischapter,wewillconcentrateonunittests,sincetheyarethebackboneofsoftware
craftingandformthevastmajorityofteststhatarewrittenbyadeveloper.
Testingisanart,anartthatyoudon’tlearnfrombooks,I’mafraid.Youcanlearnallthe
definitions(andyoushould),andtryandcollectasmuchknowledgeabouttestingasyou
canbutIpromiseyou,youwillbeabletotestyoursoftwareproperlyonlywhenyouhave
doneitforlongenoughinthefield.
Whenyouarehavingtroublerefactoringabitofcode,becauseeverylittlethingyoutouch
makesatestblowup,youlearnhowtowritelessrigidandlimitingtests,whichstillverify
thecorrectnessofyourcodebut,atthesametime,allowyouthefreedomandjoytoplay
withit,toshapeitasyouwant.
Whenyouarebeingcalledtoooftentofixunexpectedbugsinyourcode,youlearnhow
towritetestsmorethoroughly,howtocomeupwithamorecomprehensivelistofedge
cases,andstrategiestocopewiththembeforetheyturnintobugs.
Whenyouarespendingtoomuchtimereadingtestsandtryingtorefactortheminorderto
changeasmallfeatureinthecode,youlearntowritesimpler,shorter,andbetterfocused
tests.
Icouldgoonwiththiswhenyou…youlearn…,butIguessyougetthepicture.Youneed
togetyourhandsdirtyandbuildexperience.Mysuggestion?Studythetheoryasmuchas
youcan,andthenexperimentusingdifferentapproaches.Also,trytolearnfrom
experiencedcoders;it’sveryeffective.
WOW! eBook
www.wowebook.org
Theanatomyofatest
Beforeweconcentrateonunittests,let’sseewhatatestis,andwhatitspurposeis.
Atestisapieceofcodewhosepurposeistoverifysomethinginoursystem.Itmaybe
thatwe’recallingafunctionpassingtwointegers,thatanobjecthasapropertycalled
donald_duck,orthatwhenyouplaceanorderonsomeAPI,afteraminuteyoucanseeit
dissectedintoitsbasicelements,inthedatabase.
Atestistypicallycomprisedofthreesections:
Preparation:Thisiswhereyousetupthescene.Youprepareallthedata,the
objects,theservicesyouneedintheplacesyouneedthemsothattheyarereadytobe
used.
Execution:Thisiswhereyouexecutethebitoflogicthatyou’recheckingagainst.
Youperformanactionusingthedataandtheinterfacesyouhavesetupinthe
preparationphase.
Verification:Thisiswhereyouverifytheresultsandmakesuretheyareaccordingto
yourexpectations.Youcheckthereturnedvalueofafunction,orthatsomedataisin
thedatabase,someisnot,somehaschanged,arequesthasbeenmade,somethinghas
happened,amethodhasbeencalled,andsoon.
WOW! eBook
www.wowebook.org
Testingguidelines
Likesoftware,testscanbegoodorbad,withthewholerangeofshadesinthemiddle.In
ordertowritegoodtests,herearesomeguidelines:
Keepthemassimpleaspossible:It’sokaytoviolatesomegoodcodingrules,such
ashardcodingvaluesorduplicatingcode.Testsneedfirstandforemosttobeas
readableaspossibleandeasytounderstand.Whentestsarehardtoreador
understand,youcanneverbesureiftheyareactuallymakingsureyourcodeis
performingcorrectly.
Testsshouldverifyonethingandonethingonly:It’sveryimportantthatyoukeep
themshortandcontained.It’sperfectlyfinetowritemultipleteststoexerciseasingle
objectorfunction.Justmakesurethateachtesthasoneandonlyonepurpose.
Testsshouldnotmakeanyunnecessaryassumptionwhenverifyingdata:Thisis
trickytounderstandatfirst,butsayyouaretestingthereturnvalueofafunctionand
itisanunorderedlistofnumbers(like[2,3,1]).Iftheorderinthatlistisrandom,
inthetestyoumaybetemptedtosortitandcompareitwith[1,2,3].Ifyoudo,
youwillintroduceanextraassumptionontheorderingoftheresultofyourfunction
call,andthisisbadpractice.Youshouldalwaysfindawaytoverifythingswithout
introducinganyassumptionsoranyfeaturethatdoesn’tbelongintheusecaseyou’re
describingwithyourtest.
Testsshouldexercisethewhat,ratherthanthehow:Testsshouldfocuson
checkingwhatafunctionissupposedtodo,ratherthanhowitisdoingit.For
example,focusonthefactthatit’scalculatingthesquarerootofanumber(thewhat),
insteadofonthefactthatitiscallingmath.sqrttodoit(thehow).Unlessyou’re
writingperformancetestsoryouhaveaparticularneedtoverifyhowacertainaction
isperformed,trytoavoidthistypeoftestingandfocusonthewhat.Testingthehow
leadstorestrictivetestsandmakesrefactoringhard.Moreover,thetypeoftestyou
havetowritewhenyouconcentrateonthehowismorelikelytodegradethequality
ofyourtestingcodebasewhenyouamendyoursoftwarefrequently(moreonthis
later).
Testsshouldassumetheleastpossibleinthepreparationphase:Sayyouhave10
teststhatarecheckinghowadatastructureismanipulatedbyafunction.Andlet’s
saythisdatastructureisadictwithfivekey/valuepairs.Ifyouputthecompletedict
ineachtest,themomentyouhavetochangesomethinginthatdict,youalsohaveto
amendalltentests.Ontheotherhand,ifyoustripdownthetestdataasmuchasyou
can,youwillfindthat,mostofthetime,it’spossibletohavethemajorityoftests
checkingonlyapartialversionofthedata,andonlyafewrunningwithafullversion
ofit.Thismeansthatwhenyouneedtochangeyourdata,youwillhavetoamend
onlythoseteststhatareactuallyexercisingit.
Testshouldrunasfastaspossible:Agoodtestcodebasecouldendupbeingmuch
longerthanthecodebeingtesteditself.Itvariesaccordingtothesituationandthe
developerbutwhateverthelength,you’llenduphavinghundreds,ifnotthousands,
ofteststorun,whichmeansthefastertheyrun,thefasteryoucangetbacktowriting
code.WhenusingTDD,forexample,youruntestsveryoften,sospeedisessential.
WOW! eBook
www.wowebook.org
Testsshoulduseuptheleastpossibleamountofresources:Thereasonforthisis
thateverydeveloperwhochecksoutyourcodeshouldbeabletorunyourtests,no
matterhowpowerfultheirboxis.Itcouldbeaskinnyvirtualmachineoraneglected
Jenkinsbox,yourtestsshouldrunwithoutchewinguptoomanyresources.
Note
AJenkinsboxisamachinethatrunsJenkins,softwarethatiscapableof,amongst
manyotherthings,runningyourtestsautomatically.Jenkinsisfrequentlyusedin
companieswheredevelopersusepracticeslikecontinuousintegration,extreme
programming,andsoon.
WOW! eBook
www.wowebook.org
Unittesting
Nowthatyouhaveanideaaboutwhattestingisandwhyweneedit,let’sfinallyintroduce
thedeveloper’sbestfriend:theunittest.
Beforeweproceedwiththeexamples,allowmetospendsomewordsofcaution:I’lltryto
giveyouthefundamentalsaboutunittesting,butIdon’tfollowanyparticularschoolof
thoughtormethodologytotheletter.Overtheyears,Ihavetriedmanydifferenttesting
approaches,eventuallycomingupwithmyownwayofdoingthings,whichisconstantly
evolving.ToputitasBruceLeewouldhave:
“Absorbwhatisuseful,discardwhatisuselessandaddwhatisspecificallyyour
own”.
Writingaunittest
Inordertoexplainhowtowriteaunittest,let’shelpourselveswithasimplesnippet:
data.py
defget_clean_data(source):
data=load_data(source)
cleaned_data=clean_data(data)
returncleaned_data
Thefunctionget_clean_dataisresponsibleforgettingdatafromsource,cleaningit,and
returningittothecaller.Howdowetestthisfunction?
Onewayofdoingthisistocallitandthenmakesurethatload_datawascalledoncewith
sourceasitsonlyargument.Thenwehavetoverifythatclean_datawascalledonce,
withthereturnvalueofload_data.And,finally,wewouldneedtomakesurethatthe
returnvalueofclean_dataiswhatisreturnedbytheget_clean_datafunctionaswell.
Inordertodothis,weneedtosetupthesourceandrunthiscode,andthismaybea
problem.Oneofthegoldenrulesofunittestingisthatanythingthatcrossesthe
boundariesofyourapplicationneedstobesimulated.Wedon’twanttotalktoarealdata
source,andwedon’twanttoactuallyrunrealfunctionsiftheyarecommunicatingwith
anythingthatisnotcontainedinourapplication.Afewexampleswouldbeadatabase,a
searchservice,anexternalAPI,afileinthefilesystem,andsoon.
Weneedtheserestrictionstoactasashield,sothatwecanalwaysrunourtestssafely
withoutthefearofdestroyingsomethinginarealdatasource.
Anotherreasonisthatitmaybequitedifficultforasingledevelopertoreproducethe
wholearchitectureontheirbox.Itmayrequirethesettingupofdatabases,APIs,services,
filesandfolders,andsoonandsoforth,andthiscanbedifficult,timeconsuming,or
sometimesnotevenpossible.
Note
Verysimplyput,anapplicationprogramminginterface(API)isasetoftoolsfor
WOW! eBook
www.wowebook.org
buildingsoftwareapplications.AnAPIexpressesasoftwarecomponentintermsofits
operations,inputsandoutputs,andunderlyingtypes.Forexample,ifyoucreateasoftware
thatneedstointerfacewithadataproviderservice,it’sverylikelythatyouwillhavetogo
throughtheirAPIinordertogainaccesstothedata.
Therefore,inourunittests,weneedtosimulateallthosethingsinsomeway.Unittests
needtoberunbyanydeveloperwithouttheneedforthewholesystemtobesetupon
theirbox.
Adifferentapproach,whichIalwaysfavorwhenit’spossibletodoso,istosimulate
entitieswithoutusingfakeobjects,butusingspecialpurposetestobjectsinstead.For
example,ifyourcodetalkstoadatabase,insteadoffakingallthefunctionsandmethods
thattalktothedatabaseandprogrammingthefakeobjectssothattheyreturnwhatthereal
oneswould,I’dmuchratherprefertospawnatestdatabase,setupthetablesanddataI
need,andthenpatchtheconnectionsettingssothatmytestsarerunningrealcode,against
thetestdatabase,therebydoingnoharmatall.In-memorydatabasesareexcellentoptions
forthesecases.
Note
Oneoftheapplicationsthatallowyoutospawnadatabasefortesting,isDjango.Within
thedjango.testpackageyoucanfindseveraltoolsthathelpyouwriteyourtestssothat
youwon’thavetosimulatethedialogwithadatabase.Bywritingteststhisway,youwill
alsobeabletocheckontransactions,encodings,andallotherdatabaserelatedaspectsof
programming.Anotheradvantageofthisapproachconsistsintheabilityofchecking
againstthingsthatcanchangefromonedatabasetoanother.
Sometimes,though,it’sstillnotpossible,andweneedtousefakes,thereforelet’stalk
aboutthem.
Mockobjectsandpatching
Firstofall,inPython,thesefakeobjectsarecalledmocks.Uptoversion3.3,themock
librarywasathird-partylibrarythatbasicallyeveryprojectwouldinstallviapipbut,from
version3.3,ithasbeenincludedinthestandardlibraryundertheunittestmodule,and
rightfullyso,givenitsimportanceandhowwidespreaditis.
Theactofreplacingarealobjectorfunction(oringeneral,anypieceofdatastructure)
withamock,iscalledpatching.Themocklibraryprovidesthepatchtool,whichcanact
asafunctionorclassdecorator,andevenasacontextmanager(moreonthisinChapter8,
TheEdges–GUIsandScripts),thatyoucanusetomockthingsout.Onceyouhave
replacedeverythingyouneednottorun,withsuitablemocks,youcanpasstothesecond
phaseofthetestandrunthecodeyouareexercising.Aftertheexecution,youwillbeable
tocheckthosemockstoverifythatyourcodehasworkedcorrectly.
Assertions
Theverificationphaseisdonethroughtheuseofassertions.Anassertionisafunction(or
method)thatyoucanusetoverifyequalitybetweenobjects,aswellasotherconditions.
Whenaconditionisnotmet,theassertionwillraiseanexceptionthatwillmakeyourtest
WOW! eBook
www.wowebook.org
fail.Youcanfindalistofassertionsintheunittestmoduledocumentation,andtheir
correspondingPythonicversioninthenosethird-partylibrary,whichprovidesafew
advantagesoverthesheerunittestmodule,startingfromanimprovedtestdiscovery
strategy(whichisthewayatestrunnerdetectsanddiscoversthetestsinyourapplication).
Aclassicunittestexample
Mocks,patches,andassertionsarethebasictoolswe’llbeusingtowritetests.So,finally,
let’sseeanexample.I’mgoingtowriteafunctionthattakesalistofintegersandfilters
outallthosewhicharen’tpositive.
filter_funcs.py
deffilter_ints(v):
return[numfornuminvifis_positive(num)]
defis_positive(n):
returnn>0
Intheprecedingexample,wedefinethefilter_intsfunction,whichbasicallyusesalist
comprehensiontoretainallthenumbersinvthatarepositive,discardingzerosand
negativeones.Ihope,bynow,anyfurtherexplanationofthecodewouldbeinsulting.
Whatisinteresting,though,istostartthinkingabouthowwecantestit.Well,howabout
wecallfilter_intswithalistofnumbersandwemakesurethatis_positiveiscalled
foreachofthem?Ofcourse,wewouldhavetotestis_positiveaswell,butIwillshow
youlateronhowtodothat.Let’swriteasimpletestforfilter_intsnow.
Note
Justtobesurewe’reonthesamepage,Iamputtingthecodeforthischapterinafolder
calledch7,whichlieswithintherootofourproject.Atthesamelevelofch7,Ihave
createdafoldercalledtests,inwhichIhaveplacedafoldercalledtest_ch7.Inthis
folderIhaveonetestfile,calledtest_filter_func.py.
Basically,withinthetestsfolder,IwillrecreatethetreestructureofthecodeI’mtesting,
prependingeverythingwithtest_.Thisway,findingtestsisreallyeasy,aswellasis
keepingthemtidy.
tests/test_ch7/test_filter_funcs.py
fromunittestimportTestCase#1
fromunittest.mockimportpatch,call#2
fromnose.toolsimportassert_equal#3
fromch7.filter_funcsimportfilter_ints#4
classFilterIntsTestCase(TestCase):#5
@patch('ch7.filter_funcs.is_positive')#6
deftest_filter_ints(self,is_positive_mock):#7
#preparation
v=[3,-4,0,5,8]
#execution
WOW! eBook
www.wowebook.org
filter_ints(v)#8
#verification
assert_equal(
[call(3),call(-4),call(0),call(5),call(8)],
is_positive_mock.call_args_list
)#9
My,ohmy,solittlecode,andyetsomuchtosay.Firstofall:#1.TheTestCaseclassisthe
baseclassthatweusetohaveacontainedentityinwhichtorunourtests.It’snotjusta
barecontainer;itprovidesyouwithmethodstowritetestsmoreeasily.
On#2,weimportpatchandcallfromtheunittest.mockmodule.patchisresponsible
forsubstitutinganobjectwithaMockinstance,therebygivingustheabilitytocheckonit
aftertheexecutionphasehasbeencompleted.callprovidesuswithanicewayof
expressinga(forexample,function)call.
On#3,youcanseethatIprefertouseassertionsfromnose,ratherthantheonesthatcome
withtheunittestmodule.Togiveyouanexample,assert_equal(...)wouldbecome
self.assertEqual(...)ifIdidn’tusenose.Idon’tenjoytypingself.foranyassertion,
ifthereisawaytoavoidit,andIdon’tparticularlyenjoycamelcase,thereforeIalways
prefertousenosetomakemyassertions.
assert_equalisafunctionthattakestwoparameters(andanoptionalthirdonethatacts
asamessage)andverifiesthattheyarethesame.Iftheyareequal,nothinghappens,butif
theydiffer,thenanAssertionErrorexceptionisraised,tellingussomethingiswrong.
WhenIwritemytests,Ialwaysputtheexpectedvalueasthefirstargument,andthereal
oneasthesecond.ThisconventionsavesmetimewhenI’mreadingtests.
On#4,weimportthefunctionwewanttotest,andthen(#5)weproceedtocreatetheclass
whereourtestswilllive.Eachmethodofthisclassstartingwithtest_,willbeinterpreted
asatest.Asyoucansee,weneedtodecoratetest_filter_intswithpatch(#6).
Understandingthispartiscrucial,weneedtopatchtheobjectwhereitisactuallyused.In
thiscase,thepathisverysimple:ch7.filter_func.is_positive.
Tip
Patchingcanbeverytricky,soIurgeyoutoreadtheWheretopatchsectioninthemock
documentation:https://docs.python.org/3/library/unittest.mock.html#where-to-patch.
Whenwedecorateafunctionusingpatch,likeinourexample,wegetanextraargument
inthetestsignature(#7),whichIliketocallasthepatchedfunctionname,plusa_mock
suffix,justtomakeitclearthattheobjecthasbeenpatched(ormockedout).).
Finally,wegettothebodyofthetest,andwehaveaverysimplepreparationphasein
whichwesetupalistwithatleastonerepresentativeofalltheintegernumbercategories
(negative,zero,andpositive).
Then,in#8,weperformtheexecutionphase,whichrunsthefilter_intsfunction,
withoutcollectingitsresults.Ifallhasgoneasexpected,thefakeis_positivefunction
musthavebeencalledwitheachoftheintegersinv.
WOW! eBook
www.wowebook.org
Wecanverifythisbycomparingalistofcallobjectstothecall_args_listattributeon
themock(#9).Thisattributeisthelistofallthecallsperformedontheobjectsinceits
creation.
Let’srunthistest.Firstofall,makesurethatyouinstallnose($pipfreezewilltellyou
ifyouhaveitalready):
$pipinstallnose
Then,changeintotherootoftheproject(mineiscalledlearning.python),andrunthe
testslikethis:
$noseteststests/test_ch7/
.
-----------------------------------------------------------Ran1testin0.006s
OK
Theoutputshowsonedot(eachdotisatest),aseparationline,andthetimetakentorun
thewholetestsuite.ItalsosaysOKattheend,whichmeansthatourtestswereall
successful.
Makingatestfail
Good,sojustforfunlet’smakeonefail.Inthetestfile,changethelastcallfromcall(8)
tocall(9),andrunthetestsagain:
$noseteststests/test_ch7/
F
============================================================
FAIL:test_filter_ints(test_filter_funcs.FilterIntsTestCase)
-----------------------------------------------------------Traceback(mostrecentcalllast):
File"/usr/lib/python3.4/unittest/mock.py",line1125,inpatched
returnfunc(*args,**keywargs)
File"/home/fab/srv/learning.python/tests/test_ch7/test_filter_funcs.py",
line21,intest_filter_ints
is_positive_mock.call_args_list
AssertionError:[call(3),call(-4),call(0),call(5),call(9)]!=[call(3),
call(-4),call(0),call(5),call(8)]
-----------------------------------------------------------Ran1testin0.008s
FAILED(failures=1)
Wow,wemadethebeastangry!Somuchwonderfulinformation,though.Thistellsyou
thatthetesttest_filter_ints(withthepathtoit),wasrunandthatitfailed(thebigFat
thetop,wherethedotwasbefore).ItgivesyouaTraceback,thattellsyouthatinthe
test_filter_funcs.pymodule,atline21,whenassertingon
is_positive_mock.call_args_list,wehaveadiscrepancy.Thetestexpectsthelistof
callstoendwithacall(9)instance,butthereallistendswithacall(8).Thisisnothing
lessthanwonderful.
Ifyouhaveatestlikethis,canyouimaginewhatwouldhappenifyourefactoredand
introducedabugintoyourfunctionbymistake?Well,yourtestswillbreak!Theywilltell
WOW! eBook
www.wowebook.org
youthatyouhavescrewedsomethingup,andhere’sthedetails.So,yougoandcheckout
whatyoubroke.
Interfacetesting
Let’saddanothertestthatchecksonthereturnedvalue.It’sanothermethodintheclass,
soIwon’treproducethewholecodeagain:
tests/test_ch7/test_filter_funcs.py
deftest_filter_ints_return_value(self):
v=[3,-4,0,-2,5,0,8,-1]
result=filter_ints(v)
assert_list_equal([3,5,8],result)
Thistestisabitdifferentfromthepreviousone.Firstly,wecannotmocktheis_positive
function,otherwisewewouldn’tbeabletocheckontheresult.Secondly,wedon’tcheck
oncalls,butonlyontheresultofthefunctionwheninputisgiven.
Ilikethistestmuchmorethanthepreviousone.Thistypeoftestiscalledaninterface
testbecauseitchecksontheinterface(thesetofinputsandoutputs)ofthefunctionwe’re
testing.Itdoesn’tuseanymocks,whichiswhyIusethistechniquemuchmorethanthe
previousone.Let’srunthenewtestsuiteandthenlet’sseewhyIlikeinterfacetesting
morethanthosewithmocks.
$noseteststests/test_ch7/
..
-----------------------------------------------------------Ran2testsin0.006s
OK
Twotestsran,allgood(Ichangedthat9backtoan8inthefirsttest,ofcourse).
Comparingtestswithandwithoutmocks
Now,let’sseewhyIdon’treallylikemocksandusethemonlywhenIhavenochoice.
Let’srefactorthecodeinthisway:
filter_funcs_refactored.py
deffilter_ints(v):
v=[numfornuminvifnum!=0]#1
return[numfornuminvifis_positive(num)]
Thecodeforis_positiveisthesameasbefore.Butthelogicinfilter_intshasnow
changedinawaythatis_positivewillneverbecalledwitha0,sincetheyareallfiltered
outin#1.Thisleadstoaninterestingresult,solet’srunthetestsagain:
$noseteststests/test_ch7/test_filter_funcs_refactored.py
F.
============================================================
FAIL:test_filter_ints(test_filter_funcs_refactored.FilterIntsTestCase)
-----------------------------------------------------------...omit…
WOW! eBook
www.wowebook.org
AssertionError:[call(3),call(-4),call(0),call(5),call(8)]!=[call(3),
call(-4),call(5),call(8)]
-----------------------------------------------------------Ran2testsin0.002s
FAILED(failures=1)
Onetestsucceededbuttheotherone,theonewiththemockedis_positivefunction,
failed.TheAssertionErrormessageshowsusthatwenowneedtoamendthelistof
expectedcalls,removingcall(0),becauseitisnolongerperformed.
Thisisnotgood.Wehavechangedneithertheinterfaceofthefunctionnoritsbehavior.
Thefunctionisstillkeepingtoitsoriginalcontract.Whatwe’vedonebytestingitwitha
mockedobjectislimitourselves.Infact,wenowhavetoamendthetestinordertousethe
newlogic.
Thisisjustasimpleexamplebutitshowsoneimportantflawinthewholemock
mechanism.Youmustkeepyourmocksup-to-dateandinsyncwiththecodetheyare
replacing,otherwiseyouriskhavingissuesliketheprecedingone,orevenworse.Your
testsmaynotfailbecausetheyareusingmockedobjectsthatperformfine,butbecausethe
realones,nownotinsyncanymore,areactuallyfailing.
Sousemocksonlywhennecessary,onlywhenthereisnootherwayoftestingyour
functions.Whenyoucrosstheboundariesofyourapplicationinatest,trytousea
replacement,likeatestdatabase,orafakeAPI,andonlywhenit’snotpossible,resortto
mocks.Theyareverypowerful,butalsoverydangerouswhennothandledproperly.
So,let’sremovethatfirsttestandkeeponlythesecondone,sothatIcanshowyou
anotherissueyoucouldrunintowhenwritingtests.Thewholetestmodulenowlookslike
this:
tests/test_ch7/test_filter_funcs_final.py
fromunittestimportTestCase
fromnose.toolsimportassert_list_equal
fromch7.filter_funcsimportfilter_ints
classFilterIntsTestCase(TestCase):
deftest_filter_ints_return_value(self):
v=[3,-4,0,-2,5,0,8,-1]
result=filter_ints(v)
assert_list_equal([3,5,8],result)
Ifwerunit,itwillpass.
Abriefchatabouttriangulation.Nowletmeaskyou:whathappensifIchangemy
filter_intsfunctiontothis?
filter_funcs_triangulation.py
deffilter_ints(v):
return[3,5,8]
Ifyourunthetestsuite,thetestwehavewillstillpass!YoumaythinkI’mcrazybutI’m
showingyouthisbecauseIwanttotalkaboutaconceptcalledtriangulation,whichis
WOW! eBook
www.wowebook.org
veryimportantwhendoinginterfacetestingwithTDD.
Thewholeideaistoremovecheatingcode,orbadlyperformingcode,bypinpointingit
fromdifferentangles(likegoingtoonevertexofatrianglefromtheothertwo)inaway
thatmakesitimpossibleforourcodetocheat,andthebugisexposed.Wecansimply
modifythetestlikethis:
tests/test_ch7/test_filter_funcs_final_triangulation.py
deftest_filter_ints_return_value(self):
v1=[3,-4,0,-2,5,0,8,-1]
v2=[7,-3,0,0,9,1]
assert_list_equal([3,5,8],filter_ints(v1))
assert_list_equal([7,9,1],filter_ints(v2))
Ihavemovedtheexecutionsectionintheassertionsdirectly,andyoucanseethatwe’re
nowpinpointingourfunctionfromtwodifferentangles,therebyrequiringthatthereal
codebeinit.It’snolongerpossibleforourfunctiontocheat.
Triangulationisaverypowerfultechniquethatteachesustoalwaystrytoexerciseour
codefrommanydifferentangles,tocoverallpossibleedgecasestoexposeanyproblems.
Boundariesandgranularity
Let’snowaddatestfortheis_positivefunction.Iknowit’saone-liner,butitpresents
uswithopportunitytodiscusstwoveryimportantconcepts:boundariesandgranularity.
That0inthebodyofthefunctionisaboundary,the>intheinequalityishowwebehave
withregardstothisboundary.Typically,whenyousetaboundary,youdividethespace
intothreeareas:whatliesbeforetheboundary,aftertheboundary,andontheboundary
itself.Intheexample,beforetheboundarywefindthenegativenumbers,theboundaryis
theelement0and,aftertheboundary,wefindthepositivenumbers.Weneedtotesteach
oftheseareastobesurewe’retestingthefunctioncorrectly.So,let’sseeonepossible
solution(Iwilladdthetesttotheclass,butIwon’tshowtherepeatedcode):
tests/test_ch7/test_filter_funcs_is_positive_loose.py
deftest_is_positive(self):
assert_equal(False,is_positive(-2))#beforeboundary
assert_equal(False,is_positive(0))#ontheboundary
assert_equal(True,is_positive(2))#aftertheboundary
Youcanseethatweareexercisingonenumberforeachdifferentareaaroundthe
boundary.Doyouthinkthistestisgood?Thinkaboutitforaminutebeforereadingon.
Theanswerisno,thistestisnotgood.Notgoodenough,anyway.IfIchangethebodyof
theis_positivefunctiontoreadreturnn>1,Iwouldexpectmytesttofail,butit
won’t.-2isstillFalse,aswellas0,and2isstillTrue.Whydoesthathappen?Itis
becausewehaven’ttakengranularityproperlyintoaccount.We’redealingwithintegers,
sowhatistheminimumgranularitywhenwemovefromoneintegertothenextone?It’s
1.Therefore,whenwesurroundtheboundary,takingallthreeareasintoaccountisnot
enough.Weneedtodoitwiththeminimumpossiblegranularity.Let’schangethetest:
WOW! eBook
www.wowebook.org
tests/test_ch7/test_filter_funcs_is_positive_correct.py
deftest_is_positive(self):
assert_equal(False,is_positive(-1))
assert_equal(False,is_positive(0))
assert_equal(True,is_positive(1))
Ah,nowit’sbetter.Nowifwechangethebodyofis_positivetoreadreturnn>1,the
thirdassertionwillfail,whichiswhatwewant.Canyouthinkofabettertest?
tests/test_ch7/test_filter_funcs_is_positive_better.py
deftest_is_positive(self):
assert_equal(False,is_positive(0))
forninrange(1,10**4):
assert_equal(False,is_positive(-n))
assert_equal(True,is_positive(n))
Thistestisevenbetter.Wetestthefirsttenthousandintegers(bothpositiveandnegative)
and0.Itbasicallyprovidesuswithabettercoveragethanjusttheoneacrosstheboundary.
So,keepthisinmind.Zoomcloselyaroundeachboundarywithminimalgranularity,but
trytoexpandaswell,findingagoodcompromisebetweenoptimalcoverageand
executionspeed.Wewouldlovetocheckthefirstbillionintegers,butwecan’twaitdays
forourteststorun.
Amoreinterestingexample
Okay,thiswasasgentleanintroductionasIcouldgiveyou,solet’smoveontosomething
moreinteresting.Let’swriteandtestafunctionthatflattensanesteddictionarystructure.
Foracoupleofyears,IhaveworkedverycloselywithTwitterandFacebookAPIs.
Handlingsuchhumongousdatastructuresisnoteasy,especiallysincethey’reoftendeeply
nested.Itturnsoutthatit’smucheasiertoflattentheminawaythatyoucanworkon
themwithoutlosingtheoriginalstructuralinformation,andthenrecreatethenested
structurefromtheflatone.Togiveyouanexample,wewantsomethinglikethis:
data_flatten.py
nested={
'fullname':'Alessandra',
'age':41,
'phone-numbers':['+447421234567','+447423456789'],
'residence':{
'address':{
'first-line':'AlexandraRd',
'second-line':'',
},
'zip':'N80PP',
'city':'London',
'country':'UK',
},
}
flat={
'fullname':'Alessandra',
'age':41,
WOW! eBook
www.wowebook.org
'phone-numbers':['+447421234567','+447423456789'],
'residence.address.first-line':'AlexandraRd',
'residence.address.second-line':'',
'residence.zip':'N80PP',
'residence.city':'London',
'residence.country':'UK',
}
Astructurelikeflatismuchsimplertomanipulate.Beforewritingtheflattener,let’s
makesomeassumptions:thekeysarestrings,weleaveeverydatastructureasitisunless
it’sadictionary,inwhichcaseweflattenit,weusethedotasseparator,butwewanttobe
abletopassadifferentonetoourfunction.Here’sthecode:
data_flatten.py
defflatten(data,prefix='',separator='.'):
"""Flattensanesteddictstructure."""
ifnotisinstance(data,dict):
return{prefix:data}ifprefixelsedata
result={}
for(key,value)indata.items():
result.update(
flatten(
value,
_get_new_prefix(prefix,key,separator),
separator=separator))
returnresult
def_get_new_prefix(prefix,key,separator):
return(separator.join((prefix,str(key)))
ifprefixelsestr(key))
Theprecedingexampleisnotdifficult,butalsonottrivialsolet’sgothroughit.Atfirst,
wecheckifdataisadictionary.Ifit’snotadictionary,thenit’sdatathatdoesn’tneedto
beflattened;therefore,wesimplyreturneitherdataor,ifprefixisnotanemptystring,a
dictionarywithonekey/valuepair:prefix/data.
Ifinsteaddataisadict,weprepareanemptyresultdicttoreturn,thenweparsethelist
ofdata‘sitems,which,atI’msureyouwillremember,are2-tuples(key,value).Foreach
(key,value)pair,werecursivelycallflattenonthem,andweupdatetheresultdictwith
what’sreturnedbythatcall.Recursionisexcellentwhenrunningthroughnested
structures.
Ataglance,canyouunderstandwhatthe_get_new_prefixfunctiondoes?Let’susethe
inside-outtechniqueonceagain.Iseeaternaryoperatorthatreturnsthestringifiedkey
whenprefixisanemptystring.Ontheotherhand,whenprefixisanon-emptystring,
weusetheseparatortojointheprefixwiththestringifiedversionofkey.Noticethat
thebracesinsidethecalltojoinaren’tredundant,weneedthem.Canyoufigureoutwhy?
Let’swriteacoupleoftestsforthisfunction:
tests/test_ch7/test_data_flatten.py
#...importsomitted…
WOW! eBook
www.wowebook.org
classFlattenTestCase(TestCase):
deftest_flatten(self):
test_cases=[
({'A':{'B':'C','D':[1,2,3],'E':{'F':'G'}},
'H':3.14,
'J':['K','L'],
'M':'N'},
{'A.B':'C',
'A.D':[1,2,3],
'A.E.F':'G',
'H':3.14,
'J':['K','L'],
'M':'N'}),
(0,0),
('Hello','Hello'),
({'A':None},{'A':None}),
]
for(nested,flat)intest_cases:
assert_equal(flat,flatten(nested))
deftest_flatten_custom_separator(self):
nested={'A':{'B':{'C':'D'}}}
assert_equal(
{'A#B#C':'D'},flatten(nested,separator='#'))
Let’sstartfromtest_flatten.Idefinedalistof2-tuples(nested,flat),eachofwhich
representsatestcase(Ihighlightednestedtoeasereading).Ihaveonebigdictwiththree
levelsofnesting,andthensomesmallerdatastructuresthatwon’tchangewhenpassedto
theflattenfunction.Thesetestcasesareprobablynotenoughtocoveralledgecases,but
theyshouldgiveyouagoodideaofhowyoucouldstructureatestlikethis.Withasimple
forloop,Icyclethrougheachtestcaseandassertthattheresultofflatten(nested)is
equaltoflat.
Tip
Onethingtosayaboutthisexampleisthat,whenyourunit,itwillshowyouthattwotests
havebeenrun.Thisisactuallynotcorrectbecauseeveniftechnicallytherewereonlytwo
testsrunning,inoneofthemwehavemultipletestcases.Itwouldbenicertohavethem
runinawaythattheywererecognizedasseparate.Thisispossiblethroughtheuseof
librariessuchasnose-parameterized,whichIencourageyoutocheckout.It’son
https://pypi.python.org/pypi/nose-parameterized.
Ialsoprovidedasecondtesttomakesurethecustomseparatorfeatureworked.Asyou
cansee,Iusedonlyonedatastructure,whichismuchsmaller.Wedon’tneedtogobig
again,nortotestotheredgecases.Remember,testsshouldmakesureofonethingandone
thingonly,andtest_flatten_custom_separatorjusttakescareofverifyingwhetheror
notwecanfeedtheflattenfunctionadifferentseparator.
IcouldkeepblatheringonabouttestsforaboutanotherbookifonlyIhadthespace,but
unfortunately,weneedtostophere.Ihaven’ttoldyouaboutdoctests(testswritteninthe
documentationusingaPythoninteractiveshellstyle),andaboutanotherhalfamillion
WOW! eBook
www.wowebook.org
thingsthatcouldbesaidaboutthissubject.You’llhavetodiscoverthatforyourself.
Takealookatthedocumentationfortheunittestmodule,thenoseandnoseparameterizedlibraries,andpytest(http://pytest.org/),andyouwillbefine.Inmy
experience,mockingandpatchingseemtobequitehardtogetagoodgraspoffor
developerswhoarenewtothem,soallowyourselfalittletimetodigestthesetechniques.
Tryandlearnthemgradually.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Test-drivendevelopment
Let’stalkbrieflyabouttest-drivendevelopmentorTDD.Itisamethodologythatwas
rediscoveredbyKentBeck,whowroteTestDrivenDevelopmentbyExample,Addison
Wesley–2002,whichIencourageyoutocheckoutifyouwanttolearnaboutthe
fundamentalsofthissubject,whichI’mquiteobsessedwith.
TDDisasoftwaredevelopmentmethodologythatisbasedonthecontinuous
repetitionofaveryshortdevelopmentcycle.
Atfirst,thedeveloperwritesatest,andmakesitrun.Thetestissupposedtochecka
featurethatisnotyetpartofthecode.Maybeisanewfeaturetobeadded,orsomething
toberemovedoramended.Runningthetestwillmakeitfailand,becauseofthis,this
phaseiscalledRed.
Whenthetesthasfailed,thedeveloperwritestheminimalamountofcodetomakeitpass.
Whenrunningthetestsucceeds,wehavetheso-calledGreenphase.Inthisphase,itis
okaytowritecodethatcheats,justtomakethetestpass(that’swhyyouwouldthenuse
triangulation).Thistechniqueiscalled,fakeit‘tilyoumakeit.
Thelastpieceofthiscycleiswherethedevelopertakescareofboththecodeandthetests
(inseparatetimes)andrefactorsthemuntiltheyareinthedesiredstate.Thislastphaseis
calledRefactor.
TheTDDmantrathereforerecites,Red-Green-Refactor.
Atfirst,itfeelsreallyweirdtowritetestsbeforethecode,andImustconfessittookmea
whiletogetusedtoit.Ifyousticktoit,though,andforceyourselftolearnthisslightly
counter-intuitivewayofworking,atsomepointsomethingalmostmagicalhappens,and
youwillseethequalityofyourcodeincreaseinawaythatwouldn’tbepossible
otherwise.
Whenyouwriteyourcodebeforethetests,youhavetotakecareofwhatthecodehasto
doandhowithastodoit,bothatthesametime.Ontheotherhand,whenyouwritetests
beforethecode,youcanconcentrateonthewhatpartalone,whileyouwritethem.When
youwritethecodeafterwards,youwillmostlyhavetotakecareofhowthecodehasto
implementwhatisrequiredbythetests.Thisshiftinfocusallowsyourmindto
concentrateonthewhatandhowpartsinseparatemoments,yieldingabrainpowerboost
thatwillsurpriseyou.
Thereareseveralotherbenefitsthatcomefromtheadoptionofthistechnique:
Youwillrefactorwithmuchmoreconfidence:Becausewhenyoutouchyourcode
youknowthatifyouscrewthingsup,youwillbreakatleastonetest.Moreover,you
willbeabletotakecareofthearchitecturaldesignintherefactorphase,where
havingteststhatactasguardianswillallowyoutoenjoymassagingthecodeuntilit
reachesastatethatsatisfiesyou.
Thecodewillbemorereadable:Thisiscrucialinourtime,whencodingisasocial
WOW! eBook
www.wowebook.org
activityandeveryprofessionaldeveloperspendsmuchmoretimereadingcodethan
writingit.
Thecodewillbemoreloose-coupledandeasiertotestandmaintain:Thisis
simplybecausewritingthetestsfirstforcesyoutothinkmoredeeplyaboutits
structure.
Writingtestsfirstrequiresyoutohaveabetterunderstandingofthebusiness
requirements:Thisisfundamentalindeliveringwhatwasactuallyaskedfor.Ifyour
understandingoftherequirementsislackinginformation,you’llfindwritingatest
extremelychallengingandthissituationactsasasentinelforyou.
Havingeverythingunittestedmeansthecodewillbeeasiertodebug:Moreover,
smalltestsareperfectforprovidingalternativedocumentation.Englishcanbe
misleading,butfivelinesofPythoninasimpletestareveryhardtobe
misunderstood.
Higherspeed:It’sfastertowritetestsandcodethanitistowritethecodefirstand
thenlosetimedebuggingit.Ifyoudon’twritetests,youwillprobablydeliverthe
codesooner,butthenyouwillhavetotrackthebugsdownandsolvethem(and,rest
assured,therewillbebugs).Thecombinedtimetakentowritethecodeandthen
debugitisusuallylongerthanthetimetakentodevelopthecodewithTDD,where
havingtestsrunningbeforethecodeiswritten,ensuringthattheamountofbugsinit
willbemuchlowerthanintheothercase.
Ontheotherhand,themainshortcomingsofthistechniqueare:
Thewholecompanyneedstobelieveinit:Otherwiseyouwillhavetoconstantly
arguewithyourboss,whowillnotunderstandwhyittakesyousolongtodeliver.
Thetruthis,itmaytakeyouabitlongertodeliverintheshortterm,butinthelong
termyougainalotwithTDD.However,itisquitehardtoseethelongtermbecause
it’snotunderournosesliketheshorttermis.Ihavefoughtbattleswithstubborn
bossesinmycareer,tobeabletocodeusingTDD.Sometimesithasbeenpainful,
butalwayswellworthit,andIhaveneverregretteditbecause,intheend,thequality
oftheresulthasalwaysbeenappreciated.
Ifyoufailtounderstandthebusinessrequirements,thiswillreflectinthetests
youwrite,andthereforeitwillreflectinthecodetoo:Thiskindofproblemis
quitehardtospotuntilyoudoUAT,butonethingthatyoucandotoreducethe
likelihoodofithappeningistopairwithanotherdeveloper.Pairingwillinevitably
requirediscussionsaboutthebusinessrequirements,andthiswillhelphavingabetter
ideaaboutthembeforethetestsarewritten.
Badlywrittentestsarehardtomaintain:Thisisafact.Testswithtoomanymocks
orwithextraassumptionsorbadlystructureddatawillsoonbecomeaburden.Don’t
letthisdiscourageyou;justkeepexperimentingandchangethewayyouwritethem
untilyoufindawaythatdoesn’trequireyouahugeamountofworkeverytimeyou
touchyourcode.
I’msopassionateaboutTDDthatwhenIinterviewforajob,Ialwaysaskifthecompany
I’mabouttojoinadoptsit.Iftheanswerisno,it’skindofadeal-breakerforme.I
encourageyoutocheckitoutanduseit.Useituntilyoufeelsomethingclickinginyour
WOW! eBook
www.wowebook.org
mind.Youwon’tregretit,Ipromise.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Exceptions
EventhoughIhaven’tformallyintroducedthemtoyou,bynowIexpectyoutoatleast
haveavagueideaofwhatanexceptionis.Inthepreviouschapters,we’veseenthatwhen
aniteratorisexhausted,callingnextonitraisesaStopIterationexception.We’vemet
IndexErrorwhenwetriedaccessingalistatapositionthatwasoutsidethevalidrange.
We’vealsometAttributeErrorwhenwetriedaccessinganattributeonanobjectthat
didn’thaveit,andKeyErrorwhenwedidthesamewithakeyandadictionary.We’ve
alsojustmetAssertionErrorwhenrunningtests.
Now,thetimehascomeforustotalkaboutexceptions.
Sometimes,eventhoughanoperationorapieceofcodeiscorrect,thereareconditionsin
whichsomethingmaygowrong.Forexample,ifwe’reconvertinguserinputfromstring
toint,theusercouldaccidentallytypealetterinplaceofadigit,makingitimpossiblefor
ustoconvertthatvalueintoanumber.Whendividingnumbers,wemaynotknowin
advanceifwe’reattemptingadivisionbyzero.Whenopeningafile,itcouldbemissing
orcorrupted.
Whenanerrorisdetectedduringexecution,itiscalledanexception.Exceptionsarenot
necessarilylethal;infact,we’veseenthatStopIterationisdeeplyintegratedinPython
generatoranditeratormechanisms.Normally,though,ifyoudon’ttakethenecessary
precautions,anexceptionwillcauseyourapplicationtobreak.Sometimes,thisisthe
desiredbehaviorbutinothercases,wewanttopreventandcontrolproblemssuchas
these.Forexample,wemayalerttheuserthatthefilethey’retryingtoopeniscorrupted
orthatitismissingsothattheycaneitherfixitorprovideanotherfile,withouttheneed
fortheapplicationtodiebecauseofthisissue.Let’sseeanexampleofafewexceptions:
exceptions/first.example.py
>>>gen=(nforninrange(2))
>>>next(gen)
0
>>>next(gen)
1
>>>next(gen)
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
StopIteration
>>>print(undefined_var)
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
NameError:name'undefined_var'isnotdefined
>>>mylist=[1,2,3]
>>>mylist[5]
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
IndexError:listindexoutofrange
>>>mydict={'a':'A','b':'B'}
>>>mydict['c']
Traceback(mostrecentcalllast):
WOW! eBook
www.wowebook.org
File"<stdin>",line1,in<module>
KeyError:'c'
>>>1/0
Traceback(mostrecentcalllast):
File"<stdin>",line1,in<module>
ZeroDivisionError:divisionbyzero
Asyoucansee,thePythonshellisquiteforgiving.WecanseetheTraceback,sothatwe
haveinformationabouttheerror,buttheprogramdoesn’tdie.Thisisaspecialbehavior,a
regularprogramorascriptwouldnormallydieifnothingweredonetohandleexceptions.
Tohandleanexception,Pythongivesyouthetrystatement.Whathappenswhenyou
enterthetryclauseisthatPythonwillwatchoutforoneormoredifferenttypesof
exceptions(accordingtohowyouinstructit),andiftheyareraised,itwillallowyouto
react.Thetrystatementiscomprisedofthetryclause,whichopensthestatement;oneor
moreexceptclauses(alloptional)thatdefinewhattodowhenanexceptioniscaught;an
elseclause(optional),whichisexecutedwhenthetryclauseisexitedwithoutany
exceptionraised;andafinallyclause(optional),whosecodeisexecutedregardlessof
whateverhappenedintheotherclauses.Thefinallyclauseistypicallyusedtocleanup
resources.Mindtheorder,it’simportant.Also,trymustbefollowedbyatleastone
exceptclauseorafinallyclause.Let’sseeanexample:
exceptions/try.syntax.py
deftry_syntax(numerator,denominator):
try:
print('Inthetryblock:{}/{}'
.format(numerator,denominator))
result=numerator/denominator
exceptZeroDivisionErroraszde:
print(zde)
else:
print('Theresultis:',result)
returnresult
finally:
print('Exiting')
print(try_syntax(12,4))
print(try_syntax(11,0))
Theprecedingexampledefinesasimpletry_syntaxfunction.Weperformthedivisionof
twonumbers.WearepreparedtocatchaZeroDivisionErrorexceptionifwecallthe
functionwithdenominator=0.Initially,thecodeentersthetryblock.Ifdenominatoris
not0,resultiscalculatedandtheexecution,afterleavingthetryblock,resumesinthe
elseblock.Weprintresultandreturnit.Takealookattheoutputandyou’llnoticethat
justbeforereturningresult,whichistheexitpointofthefunction,Pythonexecutesthe
finallyclause.
Whendenominatoris0,thingschange.Weentertheexceptblockandprintzde.The
elseblockisn’texecutedbecauseanexceptionwasraisedinthetryblock.Before
(implicitly)returningNone,westillexecutethefinallyblock.Takealookattheoutput
andseeifitmakessensetoyou:
WOW! eBook
www.wowebook.org
$pythonexceptions/try.syntax.py
Inthetryblock:12/4
Theresultis:3.0
Exiting
3.0
Inthetryblock:11/0
divisionbyzero
Exiting
None
Whenyouexecuteatryblock,youmaywanttocatchmorethanoneexception.For
example,whentryingtodecodeaJSONobject,youmayincurintoValueErrorfor
malformedJSON,orTypeErrorifthetypeofthedatayou’refeedingtojson.loads()is
notastring.Inthiscase,youmaystructureyourcodelikethis:
exceptions/json.example.py
importjson
json_data='{}'
try:
data=json.loads(json_data)
except(ValueError,TypeError)ase:
print(type(e),e)
ThiscodewillcatchbothValueErrorandTypeError.Trychangingjson_data='{}'to
json_data=2orjson_data='{{',andyou’llseethedifferentoutput.
Note
JSONstandsforJavaScriptObjectNotationandit’sanopenstandardformatthatuses
human-readabletexttotransmitdataobjectsconsistingofkey/valuepairs.It’san
exchangeformatwidelyusedwhenmovingdataacrossapplications,especiallywhendata
needstobetreatedinalanguageorplatform-agnosticway.
Ifyouwanttohandlemultipleexceptionsdifferently,youcanjustaddmoreexcept
clauses,likethis:
exceptions/multiple.except.py
try:
#somecode
exceptException1:
#reacttoException1
except(Exception2,Exception3):
#reacttoException2andException3
exceptException3:
#reacttoException3…
Keepinmindthatanexceptionishandledinthefirstblockthatdefinesthatexception
classoranyofitsbases.Therefore,whenyoustackmultipleexceptclauseslikewe’ve
justdone,makesurethatyouputspecificexceptionsatthetopandgenericonesatthe
bottom.InOOPterms,childrenontop,grandparentsatthebottom.Moreover,remember
thatonlyoneexcepthandlerisexecutedwhenanexceptionisraised.
Youcanalsowritecustomexceptions.Inordertodothat,youjusthavetoinheritfrom
WOW! eBook
www.wowebook.org
anyotherexceptionclass.Pythonbuilt-inexceptionsaretoomanytobelistedhere,soI
havetopointyoutowardstheofficialdocumentation.Oneimportantthingtoknowisthat
everyPythonexceptionderivesfromBaseException,butyourcustomexceptionsshould
neverinheritdirectlyfromthatone.Thereasonforitisthathandlingsuchanexception
willtrapalsosystem-exitingexceptionssuchasSystemExitandKeyboardInterrupt,
whichderivefromBaseException,andthiscouldleadtosevereissues.Incaseofdisaster,
youwanttobeabletoCtrl+Cyourwayoutofanapplication.
YoucaneasilysolvetheproblembyinheritingfromException,whichinheritsfrom
BaseException,butdoesn’tincludeanysystem-exitingexceptioninitschildrenbecause
theyaresiblingsinthebuilt-inexceptionshierarchy(see
https://docs.python.org/3/library/exceptions.html#exception-hierarchy).
Programmingwithexceptionscanbeverytricky.Youcouldinadvertentlysilenceout
errors,ortrapexceptionsthataren’tmeanttobehandled.Playitsafebykeepinginminda
fewguidelines:alwaysputinthetryclauseonlythecodethatmaycausetheexception(s)
thatyouwanttohandle.Whenyouwriteexceptclauses,beasspecificasyoucan,don’t
justresorttoexceptExceptionbecauseit’seasy.Useteststomakesureyourcode
handlesedgecasesinawaythatrequirestheleastpossibleamountofexceptionhandling.
Writinganexceptstatementwithoutspecifyinganyexceptionwouldcatchanyexception,
thereforeexposingyourcodetothesamerisksyouincurwhenyouderiveyourcustom
exceptionsfromBaseException.
Youwillfindinformationaboutexceptionsalmosteverywhereontheweb.Somecoders
usethemabundantly,otherssparingly(Ibelongtothelattercategory).Findyourownway
ofdealingwiththembytakingexamplesfromotherpeople’ssourcecode.There’splenty
ofinterestingprojectswhosesourcesareopen,andyoucanfindthemoneitherGitHub
(https://github.com)orBitbucket(https://bitbucket.org/).
Beforewetalkaboutprofiling,letmeshowyouanunconventionaluseofexceptions,just
togiveyousomethingtohelpyouexpandyourviewsonthem.Theyarenotjustsimply
errors.
exceptions/for.loop.py
n=100
found=False
forainrange(n):
iffound:break
forbinrange(n):
iffound:break
forcinrange(n):
if42*a+17*b+c==5096:
found=True
print(a,b,c)#799995
Theprecedingcodeisquiteacommonidiomifyoudealwithnumbers.Youhavetoiterate
overafewnestedrangesandlookforaparticularcombinationofa,b,andcthatsatisfies
acondition.Intheexample,conditionisatriviallinearequation,butimaginesomething
muchcoolerthanthat.Whatbugsmeishavingtocheckifthesolutionhasbeenfoundat
WOW! eBook
www.wowebook.org
thebeginningofeachloop,inordertobreakoutofthemasfastaswecanwhenitis.The
breakoutlogicinterfereswiththerestofthecodeandIdon’tlikeit,soIcameupwitha
differentsolutionforthis.Takealookatit,andseeifyoucanadaptittoothercasestoo.
exceptions/for.loop.py
classExitLoopException(Exception):
pass
try:
n=100
forainrange(n):
forbinrange(n):
forcinrange(n):
if42*a+17*b+c==5096:
raiseExitLoopException(a,b,c)
exceptExitLoopExceptionasele:
print(ele)#(79,99,95)
Canyouseehowmuchmoreelegantitis?Nowthebreakoutlogicisentirelyhandledwith
asimpleexceptionwhosenameevenhintsatitspurpose.Assoonastheresultisfound,
weraiseit,andimmediatelythecontrolisgiventotheexceptclausewhichhandlesit.
Thisisfoodforthought.Thisexampleindirectlyshowsyouhowtoraiseyourown
exceptions.Readupontheofficialdocumentationtodiveintothebeautifuldetailsofthis
subject.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
ProfilingPython
ThereareafewdifferentwaystoprofileaPythonapplication.Profilingmeanshavingthe
applicationrunwhilekeepingtrackofseveraldifferentparameters,likethenumberof
timesafunctioniscalled,theamountoftimespentinsideit,andsoon.Profilingcanhelp
usfindthebottlenecksinourapplication,sothatwecanimproveonlywhatisreally
slowingusdown.
Ifyoutakealookattheprofilingsectioninthestandardlibraryofficialdocumentation,
youwillseethatthereareacoupleofdifferentimplementationsofthesameprofiling
interface:profileandcProfile.
cProfileisrecommendedformostusers,it’saCextensionwithreasonable
overheadthatmakesitsuitableforprofilinglong-runningprograms
profileisapurePythonmodulewhoseinterfaceisimitatedbycProfile,butwhich
addssignificantoverheadtoprofiledprograms
Thisinterfacedoesdeterministprofiling,whichmeansthatallfunctioncalls,function
returnsandexceptioneventsaremonitored,andprecisetimingsaremadefortheintervals
betweentheseevents.Anotherapproach,calledstatisticalprofiling,randomlysamples
theeffectiveinstructionpointer,anddeduceswheretimeisbeingspent.
Thelatterusuallyinvolveslessoverhead,butprovidesonlyapproximateresults.
Moreover,becauseofthewaythePythoninterpreterrunsthecode,deterministicprofiling
doesn’taddthatasmuchoverheadasonewouldthink,soI’llshowyouasimpleexample
usingcProfilefromthecommandline.
We’regoingtocalculatePythagoreantriples(Iknow,you’vemissedthem…)usingthe
followingcode:
profiling/triples.py
defcalc_triples(mx):
triples=[]
forainrange(1,mx+1):
forbinrange(a,mx+1):
hypotenuse=calc_hypotenuse(a,b)
ifis_int(hypotenuse):
triples.append((a,b,int(hypotenuse)))
returntriples
defcalc_hypotenuse(a,b):
return(a**2+b**2)**.5
defis_int(n):#nisexpectedtobeafloat
returnn.is_integer()
triples=calc_triples(1000)
Thescriptisextremelysimple;weiterateovertheinterval[1,mx]withaandb(avoiding
repetitionofpairsbysettingb>=a)andwecheckiftheybelongtoarighttriangle.We
usecalc_hypotenusetogethypotenuseforaandb,andthen,withis_int,wecheckifit
WOW! eBook
www.wowebook.org
isaninteger,whichmeans(a,b,c)isaPythagoreantriple.Whenweprofilethisscript,we
getinformationintabularform.Thecolumnsarencalls,tottime,percall,cumtime,
percall,andfilename:lineno(function).Theyrepresenttheamountofcallswemade
toafunction,howmuchtimewespentinit,andsoon.I’lltrimacoupleofcolumnsto
savespace,soifyouruntheprofilingyourself,don’tworryifyougetadifferentresult.
$python-mcProfileprofiling/triples.py
1502538functioncallsin0.750seconds
Orderedby:standardname
ncallstottimepercallfilename:lineno(function)
5005000.4690.000triples.py:14(calc_hypotenuse)
5005000.0870.000triples.py:18(is_int)
10.0000.000triples.py:4(<module>)
10.1630.163triples.py:4(calc_triples)
10.0000.000{built-inmethodexec}
10340.0000.000{method'append'of'list'objects}
10.0000.000{method'disable'of'_lsprof.Profil…
5005000.0320.000{method'is_integer'of'float'objects}
Evenwiththislimitedamountofdata,wecanstillinfersomeusefulinformationabout
thiscode.Firstly,wecanseethatthetimecomplexityofthealgorithmwehavechosen
growswiththesquareoftheinputsize.Theamountoftimeswegetinsidetheinnerloop
bodyisexactlymx(mx+1)/2.Werunthescriptwithmx=1000,whichmeansweget
500500timesinsidetheinnerforloop.Threemainthingshappeninsidethatloop,wecall
calc_hypotenuse,wecallis_intand,iftheconditionismet,weappendtothetriples
list.
Takingalookattheprofilingreport,wenoticethatthealgorithmhasspent0.469seconds
insidecalc_hypotenuse,whichiswaymorethanthe0.087secondsspentinsideis_int,
giventhattheywerecalledthesamenumberoftimes,solet’sseeifwecanboost
calc_hypotenusealittle.
Asitturnsout,wecan.AsImentionedearlieroninthebook,thepoweroperator**is
quiteexpensive,andincalc_hypotenuse,we’reusingitthreetimes.Fortunately,wecan
easilytransformtwoofthoseintosimplemultiplications,likethis:
profiling/triples.py
defcalc_hypotenuse(a,b):
return(a*a+b*b)**.5
Thissimplechangeshouldimprovethings.Ifweruntheprofilingagain,weseethatnow
the0.469isnowdownto0.177.Notbad!Thismeansnowwe’respendingonlyabout
37%ofthetimeinsidecalc_hypotenuseaswewerebefore.
Let’sseeifwecanimproveis_intaswell,bychangingitlikethis:
profiling/triples.py
defis_int(n):
returnn==int(n)
Thisimplementationisdifferentandtheadvantageisthatitalsoworkswhennisan
integer.Alas,whenweruntheprofilingagainstit,weseethatthetimetakeninsidethe
WOW! eBook
www.wowebook.org
is_intfunctionhasgoneupto0.141seconds.Thismeansthatithasroughlydoubled,
comparedtowhatitwasbefore.Inthiscase,weneedtoreverttotheprevious
implementation.
Thisexamplewastrivial,ofcourse,butenoughtoshowyouhowonecouldprofilean
application.Havingtheamountofcallsthatareperformedagainstafunctionhelpsus
understandbetterthetimecomplexityofouralgorithms.Forexample,youwouldn’t
believehowmanycodersfailtoseethatthosetwoforloopsrunproportionallytothe
squareoftheinputsize.
Onethingtomention:dependingonwhatsystemyou’reusing,resultsmaybedifferent.
Therefore,it’squiteimportanttobeabletoprofilesoftwareonasystemthatisascloseas
possibletotheonethesoftwareisdeployedon,ifnotactuallyonthatone.
WOW! eBook
www.wowebook.org
Whentoprofile?
Profilingissupercool,butweneedtoknowwhenitisappropriatetodoit,andinwhat
measureweneedtoaddresstheresultswegetfromit.
DonaldKnuthoncesaidthatprematureoptimizationistherootofalleviland,althoughI
wouldn’thaveputitdownsodrastically,Idoagreewithhim.Afterall,whoamIto
disagreewiththemanthatgaveusTheArtofComputerProgramming,TeX,andsomeof
thecoolestalgorithmsIhaveeverstudiedwhenIwasauniversitystudent?
So,firstandforemost:correctness.Youwantyoucodetodelivertheresultcorrectly,
thereforewritetests,findedgecases,andstressyourcodeineverywayyouthinkmakes
sense.Don’tbeprotective,don’tputthingsinthebackofyourbrainforlaterbecauseyou
thinkthey’renotlikelytohappen.Bethorough.
Secondly,takecareofcodingbestpractices.Rememberreadability,extensibility,loose
coupling,modularity,anddesign.ApplyOOPprinciples:encapsulation,abstraction,single
responsibility,open/closed,andsoon.Readupontheseconcepts.Theywillopenhorizons
foryou,andtheywillexpandthewayyouthinkaboutcode.
Thirdly,refactorlikeabeast!TheBoyScoutsRulesaystoAlwaysleavethecampground
cleanerthanyoufoundit.Applythisruletoyourcode.
And,finally,whenalloftheabovehasbeentakencareof,thenandonlythen,youtake
careofprofiling.
Runyourprofilerandidentifybottlenecks.Whenyouhaveanideaofthebottlenecksyou
needtoaddress,startwiththeworstonefirst.Sometimes,fixingabottleneckcausesa
rippleeffectthatwillexpandandchangethewaytherestofthecodeworks.Sometimes
thisisonlyalittle,sometimesabitmore,accordingtohowyourcodewasdesignedand
implemented.Therefore,startwiththebiggestissuefirst.
OneofthereasonsPythonissopopularisthatitispossibletoimplementitinmany
differentways.So,ifyoufindyourselfhavingtroublesboostingupsomepartofyour
codeusingsheerPython,nothingpreventsyoufromrollingupyoursleeves,buyinga
coupleofhundredlitersofcoffee,andrewritingtheslowpieceofcodeinC.Guaranteed
tobefun!
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,weexploredtheworldoftesting,exceptions,andprofiling.
Itriedtogiveyouafairlycomprehensiveoverviewoftesting,especiallyunittesting,
whichisthekindoftestingthatadevelopermostlydoes.IhopeIhavesucceededin
channelingthemessagethattestingisnotsomethingthatisperfectlydefinedandthatyou
canlearnfromabook.Youneedtoexperimentwithitalotbeforeyougetcomfortable.Of
alltheeffortsacodermustmakeintermsofstudyandexperimentation,I’dsaytestingis
oneofthosethataremostworthit.
We’vebrieflyseenhowwecanpreventourprogramfromdyingbecauseoferrors,called
exceptions,thathappenatruntime.And,tosteerawayfromtheusualground,Ihavegiven
youanexampleofasomewhatunconventionaluseofexceptionstobreakoutofnested
forloops.That’snottheonlycase,andI’msureyou’lldiscoverothersasyougrowasa
coder.
Intheend,weverybrieflytouchedbaseonprofiling,withasimpleexampleandafew
guidelines.Iwantedtotalkaboutprofilingforthesakeofcompleteness,soatleastyou
canplayaroundwithit.
We’renowabouttoenterChapter8,TheEdges–GUIsandScripts,wherewe’regoingto
getourhandsdirtywithscriptsandGUIsand,hopefully,comeupwithsomething
interesting.
Note
IamawarethatIgaveyoualotofpointersinthischapter,withnolinksordirections.I’m
afraidthisisbychoice.Asacoder,therewon’tbeasingledayatworkwhenyouwon’t
havetolooksomethingupinadocumentationpage,inamanual,onawebsite,andsoon.
Ithinkit’svitalforacodertobeabletosearcheffectivelyfortheinformationtheyneed,
soIhopeyou’llforgivemeforthisextratraining.Afterall,it’sallforyourbenefit.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter8.TheEdges–GUIsandScripts
“Auserinterfaceislikeajoke.Ifyouhavetoexplainit,it’snotthatgood.” —MartinLeBlanc
Inthischapter,we’regoingtoworkonaprojecttogether.We’regoingtoprepareavery
simpleHTMLpagewithafewimages,andthenwe’regoingtoscrapeit,inordertosave
thoseimages.
We’regoingtowriteascripttodothis,whichwillallowustotalkaboutafewconcepts
thatI’dliketorunbyyou.We’realsogoingtoaddafewoptionstosaveimagesbasedon
theirformat,andtochoosethewaywesavethem.And,whenwe’redonewiththescript,
we’regoingtowriteaGUIapplicationthatdoesbasicallythesamething,thuskillingtwo
birdswithonestone.Havingonlyoneprojecttoexplainwillallowmetoshowawider
rangeoftopicsinthischapter.
Note
Agraphicaluserinterface(GUI)isatypeofinterfacethatallowstheusertointeract
withanelectronicdevicethroughgraphicalicons,buttonsandwidgets,asopposedtotextbasedorcommand-lineinterfaces,whichrequirecommandsortexttobetypedonthe
keyboard.Inanutshell,anybrowser,anyofficesuitesuchasLibreOffice,and,ingeneral,
anythingthatpopsupwhenyouclickonanicon,isaGUIapplication.
So,ifyouhaven’talreadydoneso,thiswouldbetheperfecttimetostartaconsoleand
positionyourselfinafoldercalledch8intherootofyourprojectforthisbook.Withinthat
folder,we’llcreatetwoPythonmodules(scrape.pyandguiscrape.py)andonestandard
folder(simple_server).Withinsimple_server,we’llwriteourHTMLpage
(index.html)insimple_server.Imageswillbestoredinch8/simple_server/img.The
structureinch8shouldlooklikethis:
$tree-A
.
├──guiscrape.py
├──scrape.py
└──simple_server
├──img
│├──owl-alcohol.png
│├──owl-book.png
│├──owl-books.png
│├──owl-ebook.jpg
│└──owl-rose.jpeg
├──index.html
└──serve.sh
Ifyou’reusingeitherLinuxorMac,youcandowhatIdoandputthecodetostartthe
HTTPserverinaserve.shfile.OnWindows,you’llprobablywanttouseabatchfile.
TheHTMLpagewe’regoingtoscrapehasthefollowingstructure:
simple_server/index.html
WOW! eBook
www.wowebook.org
<!DOCTYPEhtml>
<htmllang="en">
<head><title>CoolOwls!</title></head>
<body>
<h1>Welcometomyowlgallery</h1>
<div>
<imgsrc="img/owl-alcohol.png"height="128"/>
<imgsrc="img/owl-book.png"height="128"/>
<imgsrc="img/owl-books.png"height="128"/>
<imgsrc="img/owl-ebook.jpg"height="128"/>
<imgsrc="img/owl-rose.jpeg"height="128"/>
</div>
<p>Doyoulikemyowls?</p>
</body>
</html>
It’sanextremelysimplepage,solet’sjustnotethatwehavefiveimages,threeofwhich
arePNGsandtwoareJPGs(notethateventhoughtheyarebothJPGs,oneendswith.jpg
andtheotherwith.jpeg,whicharebothvalidextensionsforthisformat).
So,PythongivesyouaverysimpleHTTPserverforfreethatyoucanstartwiththe
followingcommand(inthesimple_serverfolder):
$python-mhttp.server8000
ServingHTTPon0.0.0.0port8000…
127.0.0.1--[31/Aug/201516:11:10]"GET/HTTP/1.1"200-
Thelastlineisthelogyougetwhenyouaccesshttp://localhost:8000,whereour
beautifulpagewillbeserved.Alternatively,youcanputthatcommandinafilecalled
serve.sh,andjustrunthatwiththiscommand(makesureit’sexecutable):
$./serve.sh
Itwillhavethesameeffect.Ifyouhavethecodeforthisbook,yourpageshouldlook
somethinglikethis:
WOW! eBook
www.wowebook.org
Feelfreetouseanyothersetofimages,aslongasyouuseatleastonePNGandoneJPG,
andthatinthesrctagyouuserelativepaths,notabsolute.Igotthoselovelyowlsfrom
https://openclipart.org/.
WOW! eBook
www.wowebook.org
Firstapproach–scripting
Now,let’sstartwritingthescript.I’llgothroughthesourceinthreesteps:importsfirst,
thentheargumentparsinglogic,andfinallythebusinesslogic.
WOW! eBook
www.wowebook.org
Theimports
scrape.py(Imports)
importargparse
importbase64
importjson
importos
frombs4importBeautifulSoup
importrequests
Goingthroughthemfromthetop,youcanseethatwe’llneedtoparsethearguments.
whichwe’llfeedtothescriptitself(argparse).Wewillneedthebase64librarytosave
theimageswithinaJSONfile(base64andjson),andwe’llneedtoopenfilesforwriting
(os).Finally,we’llneedBeautifulSoupforscrapingthewebpageeasily,andrequeststo
fetchitscontent.requestsisanextremelypopularlibraryforperformingHTTPrequests,
builttoavoidthedifficultiesandquirksofusingthestandardlibraryurllibmodule.It’s
basedonthefasturllib3third-partylibrary.
Note
WewillexploretheHTTPprotocolandrequestsmechanisminChapter10,Web
DevelopmentDoneRightso,fornow,let’sjust(simplistically)saythatweperforman
HTTPrequesttofetchthecontentofawebpage.Wecandoitprogrammaticallyusinga
librarysuchasrequests,andit’smoreorlesstheequivalentoftypingaURLinyour
browserandpressingEnter(thebrowserthenfetchesthecontentofawebpageandalso
displaysittoyou).
Ofalltheseimports,onlythelasttwodon’tbelongtothePythonstandardlibrary,butthey
aresowidelyusedthroughouttheworldthatIdarenotexcludetheminthisbook.Make
sureyouhavetheminstalled:
$pipfreeze|egrep-i"soup|requests"
beautifulsoup4==4.4.0
requests==2.7.0
Ofcourse,theversionnumbersmightbedifferentforyou.Ifthey’renotinstalled,usethis
commandtodoso:
$pipinstallbeautifulsoup4requests
Atthispoint,theonlythingthatIreckonmightconfuseyouisthebase64/jsoncouple,so
allowmetospendafewwordsonthat.
Aswesawinthepreviouschapter,JSONisoneofthemostpopularformatsfordata
exchangebetweenapplications.It’salsowidelyusedforotherpurposestoo,forexample,
tosavedatainafile.Inourscript,we’regoingtooffertheusertheabilitytosaveimages
asimagefiles,orasaJSONsinglefile.WithintheJSON,we’llputadictionarywithkeys
astheimagesnamesandvaluesastheircontent.Theonlyissueisthatsavingimagesin
thebinaryformatistricky,andthisiswherethebase64librarycomestotherescue.
Base64isaverypopularbinary-to-textencodingschemethatrepresentsbinarydatainan
WOW! eBook
www.wowebook.org
ASCIIstringformatbytranslatingitintoaradix-64representation.
Note
Theradix-64representationusesthelettersA-Z,a-z,andthedigits0-9,plusthetwo
symbols+and/foragrandtotalof64symbolsaltogether.Therefore,notsurprisingly,the
Base64alphabetismadeupofthese64symbols.
Ifyouthinkyouhaveneverusedit,thinkagain.Everytimeyousendanemailwithan
imageattachedtoit,theimagegetsencodedwithBase64beforetheemailissent.Onthe
recipientside,imagesareautomaticallydecodedintotheiroriginalbinaryformatsothat
theemailclientcandisplaythem.
WOW! eBook
www.wowebook.org
Parsingarguments
Nowthatthetechnicalitiesareoutoftheway,let’sseethesecondsectionofourscript(it
shouldbeattheendofthescrape.pymodule).
scrape.py(Argumentparsingandscrapertriggering)
if__name__=="__main__":
parser=argparse.ArgumentParser(
description='Scrapeawebpage.')
parser.add_argument(
'-t',
'--type',
choices=['all','png','jpg'],
default='all',
help='Theimagetypewewanttoscrape.')
parser.add_argument(
'-f',
'--format',
choices=['img','json'],
default='img',
help='Theformatimagesaresavedto.')
parser.add_argument(
'url',
help='TheURLwewanttoscrapeforimages.')
args=parser.parse_args()
scrape(args.url,args.format,args.type)
Lookatthatfirstline;itisaverycommonidiomwhenitcomestoscripting.Accordingto
theofficialPythondocumentation,thestring'__main__'isthenameofthescopein
whichtop-levelcodeexecutes.Amodule’s__name__issetequalto'__main__'when
readfromstandardinput,ascript,orfromaninteractiveprompt.
Therefore,ifyouputtheexecutionlogicunderthatif,theresultisthatyouwillbeableto
usethemoduleasalibraryshouldyouneedtoimportanyofthefunctionsorobjects
definedinit,becausewhenimportingitfromanothermodule,__name__won’tbe
'__main__'.Ontheotherhand,whenyourunthescriptdirectly,likewe’regoingto,
__name__willbe'__main__',sotheexecutionlogicwillrun.
Thefirstthingwedothenisdefineourparser.Iwouldrecommendusingthestandard
librarymodule,argparse,whichissimpleenoughandquitepowerful.Thereareother
optionsoutthere,butinthiscase,argparsewillprovideuswithallweneed.
Wewanttofeedourscriptthreedifferentdata:thetypeofimageswewanttosave,the
formatinwhichwewanttosavethem,andtheURLforthepagetobescraped.
ThetypecanbePNG,JPGorboth(default),whiletheformatcanbeeitherimageor
JSON,imagebeingthedefault.URListheonlymandatoryargument.
So,weaddthe-toption,allowingalsothelongversion--type.Thechoicesare'all',
'png',and'jpg'.Wesetthedefaultto'all'andweaddahelpmessage.
Wedoasimilarprocedurefortheformatargumentallowingboththeshortandlong
WOW! eBook
www.wowebook.org
syntax(-fand--format),andfinallyweaddtheurlargument,whichistheonlyonethat
isspecifieddifferentlysothatitwon’tbetreatedasanoption,butratherasapositional
argument.
Inordertoparseallthearguments,allweneedisparser.parse_args().Verysimple,
isn’tit?
Thelastlineiswherewetriggertheactuallogic,bycallingthescrapefunction,passing
alltheargumentswejustparsed.Wewillseeitsdefinitionshortly.
Thenicethingaboutargparseisthatifyoucallthescriptbypassing-h,itwillprinta
niceusagetextforyouautomatically.Let’stryitout:
$pythonscrape.py-h
usage:scrape.py[-h][-t{all,png,jpg}][-f{img,json}]url
Scrapeawebpage.
positionalarguments:
urlTheURLwewanttoscrapeforimages.
optionalarguments:
-h,--helpshowthishelpmessageandexit
-t{all,png,jpg},--type{all,png,jpg}
Theimagetypewewanttoscrape.
-f{img,json},--format{img,json}
Theformatimagesaresavedto.
Ifyouthinkaboutit,theonetrueadvantageofthisisthatwejustneedtospecifythe
argumentsandwedon’thavetoworryabouttheusagetext,whichmeanswewon’thave
tokeepitinsyncwiththearguments’definitioneverytimewechangesomething.Thisis
precious.
Here’safewdifferentwaystocallourscrape.pyscript,whichdemonstratethattypeand
formatareoptional,andhowyoucanusetheshortandlongsyntaxtousethem:
$pythonscrape.pyhttp://localhost:8000
$pythonscrape.py-tpnghttp://localhost:8000
$pythonscrape.py--type=jpg-fjsonhttp://localhost:8000
Thefirstoneisusingdefaultvaluesfortypeandformat.Thesecondonewillsaveonly
PNGimages,andthethirdonewillsaveonlyJPGs,butinJSONformat.
WOW! eBook
www.wowebook.org
Thebusinesslogic
Nowthatwe’veseenthescaffolding,let’sdivedeepintotheactuallogic(ifitlooks
intimidatingdon’tworry;we’llgothroughittogether).Withinthescript,thislogiclies
aftertheimportsandbeforetheparsing(beforetheif__name__clause):
scrape.py(Businesslogic)
defscrape(url,format_,type_):
try:
page=requests.get(url)
exceptrequests.RequestExceptionasrex:
print(str(rex))
else:
soup=BeautifulSoup(page.content,'html.parser')
images=_fetch_images(soup,url)
images=_filter_images(images,type_)
_save(images,format_)
def_fetch_images(soup,base_url):
images=[]
forimginsoup.findAll('img'):
src=img.get('src')
img_url=(
'{base_url}/{src}'.format(
base_url=base_url,src=src))
name=img_url.split('/')[-1]
images.append(dict(name=name,url=img_url))
returnimages
def_filter_images(images,type_):
iftype_=='all':
returnimages
ext_map={
'png':['.png'],
'jpg':['.jpg','.jpeg'],
}
return[
imgforimginimages
if_matches_extension(img['name'],ext_map[type_])
]
def_matches_extension(filename,extension_list):
name,extension=os.path.splitext(filename.lower())
returnextensioninextension_list
def_save(images,format_):
ifimages:
ifformat_=='img':
_save_images(images)
else:
_save_json(images)
print('Done')
else:
print('Noimagestosave.')
WOW! eBook
www.wowebook.org
def_save_images(images):
forimginimages:
img_data=requests.get(img['url']).content
withopen(img['name'],'wb')asf:
f.write(img_data)
def_save_json(images):
data={}
forimginimages:
img_data=requests.get(img['url']).content
b64_img_data=base64.b64encode(img_data)
str_img_data=b64_img_data.decode('utf-8')
data[img['name']]=str_img_data
withopen('images.json','w')asijson:
ijson.write(json.dumps(data))
Let’sstartwiththescrapefunction.Thefirstthingitdoesisfetchthepageatthegiven
urlargument.Whatevererrormayhappenwhiledoingthis,wetrapitinthe
RequestExceptionrexandweprintit.TheRequestExceptionisthebaseexception
classforalltheexceptionsintherequestslibrary.
However,ifthingsgowell,andwehaveapagebackfromtheGETrequest,thenwecan
proceed(elsebranch)andfeeditscontenttotheBeautifulSoupparser.The
BeautifulSouplibraryallowsustoparseawebpageinnotime,withouthavingtowrite
allthelogicthatwouldbeneededtofindalltheimagesinapage,whichwereallydon’t
wanttodo.It’snotaseasyasitseems,andreinventingthewheelisnevergood.Tofetch
images,weusethe_fetch_imagesfunctionandwefilterthemwith_filter_images.
Finally,wecall_savewiththeresult.
Splittingthecodeintodifferentfunctionswithmeaningfulnamesallowsustoreaditmore
easily.Evenifyouhaven’tseenthelogicofthe_fetch_images,_filter_images,and
_savefunctions,it’snothardtopredictwhattheydo,right?
_fetch_imagestakesaBeautifulSoupobjectandabaseURL.Allitdoesislooping
throughalloftheimagesfoundonthepageandfillinginthe'name'and'url'
informationabouttheminadictionary(oneperimage).Alldictionariesareaddedtothe
imageslist,whichisreturnedattheend.
Thereissometrickerygoingonwhenwegetthenameofanimage.Whatwedoissplit
theimg_url(http://localhost:8000/img/my_image_name.png)stringusing'/'asa
separator,andwetakethelastitemastheimagename.Thereisamorerobustwayof
doingthis,butforthisexampleitwouldbeoverkill.Ifyouwanttoseethedetailsofeach
step,trytobreakthislogicdownintosmallersteps,andprinttheresultofeachofthemto
helpyourselfunderstand.
Towardstheendofthebook,I’llshowyouanothertechniquetodebuginamuchmore
efficientway.
Anyway,byjustaddingprint(images)attheendofthe_fetch_imagesfunction,weget
this:
WOW! eBook
www.wowebook.org
[{'url':'http://localhost:8000/img/owl-alcohol.png','name':'owlalcohol.png'},{'url':'http://localhost:8000/img/owl-book.png','name':
'owl-book.png'},...]
Itruncatedtheresultforbrevity.Youcanseeeachdictionaryhasa'url'and'name'
key/valuepair,whichwecanusetofetch,identifyandsaveourimagesaswelike.Atthis
point,Ihearyouaskingwhatwouldhappeniftheimagesonthepagewerespecifiedwith
anabsolutepathinsteadofarelativeone,right?Goodquestion!
Theansweristhatthescriptwillfailtodownloadthembecausethislogicexpectsrelative
paths.IwasabouttoaddabitoflogictosolvethisissuewhenIthoughtthat,atthisstage,
itwouldbeaniceexerciseforyoutodoit,soI’llleaveituptoyoutofixit.
Tip
Hint:inspectthestartofthatsrcvariable.Ifitstartswith'http',thenit’sprobablyan
absolutepath.
Ihopethebodyofthe_filter_imagesfunctionisinterestingtoyou.Iwantedtoshow
youhowtocheckonmultipleextensionsbyusingamappingtechnique.
Inthisfunction,iftype_is'all',thennofilteringisrequired,sowejustreturnallthe
images.Ontheotherhand,whentype_isnot'all',wegettheallowedextensionsfrom
theext_mapdictionary,anduseittofiltertheimagesinthelistcomprehensionthatends
thefunctionbody.Youcanseethatbyusinganotherhelperfunction,
_matches_extension,Ihavemadethelistcomprehensionsimplerandmorereadable.
All_matches_extensiondoesissplitthenameoftheimagegettingitsextensionand
checkingwhetheritiswithinthelistofallowedones.Canyoufindonemicro
improvement(speed-wise)thatcouldbedonetothisfunction?
I’msurethatyou’rewonderingwhyIhavecollectedalltheimagesinthelistandthen
removedthem,insteadofcheckingwhetherIwantedtosavethembeforeaddingthemto
thelist.ThefirstreasonisthatIneeded_fetch_imagesintheGUIappasitisnow.A
secondreasonisthatcombining,fetching,andfilteringwouldproducealongerandabit
morecomplicatedfunction,andI’mtryingtokeepthecomplexityleveldown.Athird
reasonisthatthiscouldbeaniceexerciseforyoutodo.Feelslikewe’repairinghere…
Let’skeepgoingthroughthecodeandinspectthe_savefunction.Youcanseethat,when
imagesisn’tempty,thisbasicallyactsasadispatcher.Weeithercall_save_imagesor
_save_json,dependingonwhichinformationisstoredintheformat_variable.
Wearealmostdone.Let’sjumpto_save_images.Weloopontheimageslistandforeach
dictionarywefindthereweperformaGETrequestontheimageURLandsaveitscontent
inafile,whichwenameastheimageitself.Theoneimportantthingtonotehereishow
wesavethatfile.
Weuseacontextmanager,representedbythekeywordwith,todothat.Python’swith
statementsupportstheconceptofaruntimecontextdefinedbyacontextmanager.Thisis
implementedusingapairofmethods(contextmanager.__enter__()and
contextmanager.__exit__(exc_type,exc_val,exc_tb))thatallowuser-defined
WOW! eBook
www.wowebook.org
classestodefinearuntimecontextthatisenteredbeforethestatementbodyisexecuted
andexitedwhenthestatementends.
Inourcase,usingacontextmanager,inconjunctionwiththeopenfunction,givesusthe
guaranteethatifanythingbadweretohappenwhilewritingthatfile,theresources
involvedintheprocesswillbecleanedupandreleasedproperlyregardlessoftheerror.
HaveyouevertriedtodeleteafileonWindows,onlytobepresentedwithanalertthat
tellsyouthatyoucannotdeletethefilebecausethereisanotherprocessthatisholdingon
toit?We’reavoidingthatsortofveryannoyingthing.
Whenweopenafile,wegetahandlerforitand,nomatterwhathappens,wewanttobe
surewereleaseitwhenwe’redonewiththefile.Acontextmanageristhetoolweneedto
makesureofthat.
Finally,let’snowstepintothe_save_jsonfunction.It’sverysimilartothepreviousone.
Webasicallyfillinthedatadictionary.Theimagenameisthekey,andtheBase64
representationofitsbinarycontentisthevalue.Whenwe’redonepopulatingour
dictionary,weusethejsonlibrarytodumpitintheimages.jsonfile.I’llgiveyouasmall
previewofthat:
images.json(truncated)
{
"owl-ebook.jpg":"/9j/4AAQSkZJRgABAQEAMQAxAAD/2wBDAAEBAQ…
"owl-book.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAEbCAYAAAB…
"owl-books.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAElCAYAAA…
"owl-alcohol.png":"iVBORw0KGgoAAAANSUhEUgAAASwAAAEICAYA…
"owl-rose.jpeg":"/9j/4AAQSkZJRgABAQEANAA0AAD/2wBDAAEBAQ…
}
Andthat’sit!Now,beforeproceedingtothenextsection,makesureyouplaywiththis
scriptandunderstandwellhowitworks.Tryandmodifysomething,printoutintermediate
results,addanewargumentorfunctionality,orscramblethelogic.We’regoingtomigrate
itintoaGUIapplicationnow,whichwilladdalayerofcomplexitysimplybecausewe’ll
havetobuildtheGUIinterface,soit’simportantthatyou’rewellacquaintedwiththe
businesslogic:itwillallowyoutoconcentrateontherestofthecode.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Secondapproach–aGUIapplication
ThereareseverallibrariestowriteGUIapplicationsinPython.Themostfamousonesare
tkinter,wxPython,PyGTK,andPyQt.Theyallofferawiderangeoftoolsandwidgets
thatyoucanusetocomposeaGUIapplication.
TheoneI’mgoingtousefortherestofthischapteristkinter.tkinterstandsforTk
interfaceanditisthestandardPythoninterfacetotheTkGUItoolkit.BothTkandtkinter
areavailableonmostUnixplatforms,MacOSX,aswellasonWindowssystems.
Let’smakesurethattkinterisinstalledproperlyonyoursystembyrunningthis
command:
$python-mtkinter
ItshouldopenadialogwindowdemonstratingasimpleTkinterface.Ifyoucanseethat,
thenwe’regoodtogo.However,ifitdoesn’twork,pleasesearchfortkinterinthe
Pythonofficialdocumentation.Youwillfindseverallinkstoresourcesthatwillhelpyou
getupandrunningwithit.
We’regoingtomakeaverysimpleGUIapplicationthatbasicallymimicsthebehaviorof
thescriptwesawinthefirstpartofthischapter.Wewon’taddtheabilitytosaveJPGsor
PNGssingularly,butafteryou’vegonethroughthischapter,youshouldbeabletoplay
withthecodeandputthatfeaturebackinbyyourself.
So,thisiswhatwe’reaimingfor:
Gorgeous,isn’tit?Asyoucansee,it’saverysimpleinterface(thisishowitshouldlook
onUbuntu).Thereisaframe(thatis,acontainer)fortheURLfieldandtheFetchinfo
button,anotherframefortheListboxtoholdtheimagenamesandtheradiobuttonto
controlthewaywesavethem,andfinallythereisaScrape!buttonatthebottom.Wealso
haveastatusbar,whichshowsussomeinformation.
Inordertogetthislayout,wecouldjustplaceallthewidgetsonarootwindow,butthat
WOW! eBook
www.wowebook.org
wouldmakethelayoutlogicquitemessyandunnecessarilycomplicated.So,instead,we
willdividethespaceusingframesandplacethewidgetsinthoseframes.Thiswaywewill
achieveamuchnicerresult.So,thisisthedraftforthelayout:
WehaveaRootWindow,whichisthemainwindowoftheapplication.Wedivideitinto
tworows,thefirstoneinwhichweplacetheMainFrame,andthesecondoneinwhich
weplacetheStatusFrame(whichwillholdthestatusbar).TheMainFrameis
subsequentlydividedintothreerowsitself.InthefirstoneweplacetheURLFrame,
whichholdstheURLwidgets.InthesecondoneweplacetheImgFrame,whichwill
holdtheListboxandtheRadioFrame,whichwillhostalabelandtheradiobutton
widgets.Andfinallyathirdone,whichwilljustholdtheScrapebutton.
Inordertolayoutframesandwidgets,wewillusealayoutmanagercalledgrid,that
simplydividesupthespaceintorowsandcolumns,asinamatrix.
Now,allthecodeI’mgoingtowritecomesfromtheguiscrape.pymodule,soIwon’t
repeatitsnameforeachsnippet,tosavespace.Themoduleislogicallydividedintothree
sections,notunlikethescriptversion:imports,layoutlogic,andbusinesslogic.We’re
goingtoanalyzethemlinebyline,inthreechunks.
WOW! eBook
www.wowebook.org
Theimports
fromtkinterimport*
fromtkinterimportttk,filedialog,messagebox
importbase64
importjson
importos
frombs4importBeautifulSoup
importrequests
We’realreadyfamiliarwithmostofthese.Theinterestingbithereisthosefirsttwolines.
Thefirstoneisquitecommonpractice,althoughitisbadpracticeinPythontoimport
usingthestarsyntax.Youcanincurinnamecollisionsand,ifthemoduleistoobig,
importingeverythingwouldbeexpensive.
Afterthat,weimportttk,filedialog,andmessageboxexplicitly,followingthe
conventionalapproachusedwiththislibrary.ttkisthenewsetofstyledwidgets.They
behavebasicallyliketheoldones,butarecapableofdrawingthemselvescorrectly
accordingtothestyleyourOSisseton,whichisnice.
Therestoftheimportsiswhatweneedinordertocarryoutthetaskyouknowwellby
now.Notethatthereisnothingweneedtoinstallwithpipinthissecondpart,wealready
haveeverythingweneed.
WOW! eBook
www.wowebook.org
Thelayoutlogic
I’mgoingtopasteitchunkbychunksothatIcanexplainiteasilytoyou.You’llseehow
allthosepieceswetalkedaboutinthelayoutdraftarearrangedandgluedtogether.What
I’mabouttopaste,aswedidinthescriptbefore,isthefinalpartoftheguiscrape.py
module.We’llleavethemiddlepart,thebusinesslogic,forlast.
if__name__=="__main__":
_root=Tk()
_root.title('Scrapeapp')
Asyouknowbynow,weonlywanttoexecutethelogicwhenthemoduleisrundirectly,
sothatfirstlineshouldn’tsurpriseyou.
Inthelasttwolines.wesetupthemainwindow,whichisaninstanceoftheTkclass.We
instantiateitandgiveitatitle.NotethatIusetheprependingunderscoretechniqueforall
thenamesofthetkinterobjects,inordertoavoidpotentialcollisionswithnamesinthe
businesslogic.Ijustfinditcleanerlikethis,butyou’reallowedtodisagree.
_mainframe=ttk.Frame(_root,padding='5555')
_mainframe.grid(row=0,column=0,sticky=(E,W,N,S))
Here,wesetuptheMainFrame.It’sattk.Frameinstance.Weset_rootasitsparent,
andgiveitsomepadding.Thepaddingisameasureinpixelsofhowmuchspaceshould
beinsertedbetweentheinnercontentandthebordersinordertoletourlayoutbreathea
little,otherwisewehavethesardineeffect,wherewidgetsarepackedtootightly.
Thesecondlineismuchmoreinteresting.Weplacethis_mainframeonthefirstrow(0)
andfirstcolumn(0)oftheparentobject(_root).Wealsosaythatthisframeneedsto
extenditselfineachdirectionbyusingthestickyargumentwithallfourcardinal
directions.Ifyou’rewonderingwheretheycamefrom,it’sthefromtkinterimport*
magicthatbroughtthemtous.
_url_frame=ttk.LabelFrame(
_mainframe,text='URL',padding='5555')
_url_frame.grid(row=0,column=0,sticky=(E,W))
_url_frame.columnconfigure(0,weight=1)
_url_frame.rowconfigure(0,weight=1)
Next,westartbyplacingtheURLFramedown.Thistime,theparentobjectis
_mainframe,asyouwillrecallfromourdraft.ThisisnotjustasimpleFrame,butit’s
actuallyaLabelFrame,whichmeanswecansetthetextargumentandexpectarectangle
tobedrawnaroundit,withthecontentofthetextargumentwritteninthetop-leftpartofit
(checkoutthepreviouspictureifithelps).Wepositionthisframeat(0,0),andsaythatit
shouldexpandtotheleftandtotheright.Wedon’tneedtheothertwodirections.
Finally,weuserowconfigureandcolumnconfiguretomakesureitbehavescorrectly,
shoulditneedtoresize.Thisisjustaformalityinourpresentlayout.
_url=StringVar()
_url.set('http://localhost:8000')
_url_entry=ttk.Entry(
WOW! eBook
www.wowebook.org
_url_frame,width=40,textvariable=_url)
_url_entry.grid(row=0,column=0,sticky=(E,W,S,N),padx=5)
_fetch_btn=ttk.Button(
_url_frame,text='Fetchinfo',command=fetch_url)
_fetch_btn.grid(row=0,column=1,sticky=W,padx=5)
Here,wehavethecodetolayouttheURLtextboxandthe_fetchbutton.Atextboxin
thisenvironmentiscalledEntry.Weinstantiateitasusual,setting_url_frameasits
parentandgivingitawidth.Also,andthisisthemostinterestingpart,wesetthe
textvariableargumenttobe_url._urlisaStringVar,whichisanobjectthatisnow
connectedtoEntryandwillbeusedtomanipulateitscontent.Therefore,wedon’tmodify
thetextinthe_url_entryinstancedirectly,butbyaccessing_url.Inthiscase,wecall
thesetmethodonittosettheinitialvaluetotheURLofourlocalwebpage.
Weposition_url_entryat(0,0),settingallfourcardinaldirectionsforittostickto,and
wealsosetabitofextrapaddingontheleftandrightedgesbyusingpadx,whichadds
paddingonthex-axis(horizontal).Ontheotherhand,padytakescareofthevertical
direction.
Bynow,youshouldgetthateverytimeyoucallthe.gridmethodonanobject,we’re
basicallytellingthegridlayoutmanagertoplacethatobjectsomewhere,accordingto
rulesthatwespecifyasargumentsinthegrid()call.
Similarly,wesetupandplacethe_fetchbutton.Theonlyinterestingparameteris
command=fetch_url.Thismeansthatwhenweclickthisbutton,weactuallycallthe
fetch_urlfunction.Thistechniqueiscalledcallback.
_img_frame=ttk.LabelFrame(
_mainframe,text='Content',padding='9000')
_img_frame.grid(row=1,column=0,sticky=(N,S,E,W))
ThisiswhatwecalledImgFrameinthelayoutdraft.Itisplacedonthesecondrowofits
parent_mainframe.ItwillholdtheListboxandtheRadioFrame.
_images=StringVar()
_img_listbox=Listbox(
_img_frame,listvariable=_images,height=6,width=25)
_img_listbox.grid(row=0,column=0,sticky=(E,W),pady=5)
_scrollbar=ttk.Scrollbar(
_img_frame,orient=VERTICAL,command=_img_listbox.yview)
_scrollbar.grid(row=0,column=1,sticky=(S,N),pady=6)
_img_listbox.configure(yscrollcommand=_scrollbar.set)
Thisisprobablythemostinterestingbitofthewholelayoutlogic.Aswedidwiththe
_url_entry,weneedtodrivethecontentsofListboxbytyingittoavariable_images.
WesetupListboxsothat_img_frameisitsparent,and_imagesisthevariableit’stiedto.
Wealsopasssomedimensions.
Theinterestingbitcomesfromthe_scrollbarinstance.Notethat,whenweinstantiateit,
wesetitscommandto_img_listbox.yview.Thisisthefirsthalfofthecontractbetween
aListboxandaScrollbar.Theotherhalfisprovidedbythe_img_listbox.configure
method,whichsetstheyscrollcommand=_scrollbar.set.
WOW! eBook
www.wowebook.org
Byprovidingthisreciprocalbond,whenwescrollonListbox,theScrollbarwillmove
accordinglyandvice-versa,whenweoperatetheScrollbar,theListboxwillscroll
accordingly.
_radio_frame=ttk.Frame(_img_frame)
_radio_frame.grid(row=0,column=2,sticky=(N,S,W,E))
WeplacetheRadioFrame,readytobepopulated.NotethattheListboxisoccupying(0,
0)on_img_frame,theScrollbar(0,1)andtherefore_radio_framewillgoin(0,2).
_choice_lbl=ttk.Label(
_radio_frame,text="Choosehowtosaveimages")
_choice_lbl.grid(row=0,column=0,padx=5,pady=5)
_save_method=StringVar()
_save_method.set('img')
_img_only_radio=ttk.Radiobutton(
_radio_frame,text='AsImages',variable=_save_method,
value='img')
_img_only_radio.grid(
row=1,column=0,padx=5,pady=2,sticky=W)
_img_only_radio.configure(state='normal')
_json_radio=ttk.Radiobutton(
_radio_frame,text='AsJSON',variable=_save_method,
value='json')
_json_radio.grid(row=2,column=0,padx=5,pady=2,sticky=W)
Firstly,weplacethelabel,andwegiveitsomepadding.Notethatthelabelandradio
buttonsarechildrenof_radio_frame.
AsfortheEntryandListboxobjects,theRadiobuttonisalsodrivenbyabondtoan
externalvariable,whichIcalled_save_method.EachRadiobuttoninstancesetsavalue
argument,andbycheckingthevalueon_save_method,weknowwhichbuttonisselected.
_scrape_btn=ttk.Button(
_mainframe,text='Scrape!',command=save)
_scrape_btn.grid(row=2,column=0,sticky=E,pady=5)
Onthethirdrowof_mainframeweplacetheScrapebutton.Itscommandissave,which
savestheimagestobelistedinListbox,afterwehavesuccessfullyparsedawebpage.
_status_frame=ttk.Frame(
_root,relief='sunken',padding='2222')
_status_frame.grid(row=1,column=0,sticky=(E,W,S))
_status_msg=StringVar()
_status_msg.set('TypeaURLtostartscraping…')
_status=ttk.Label(
_status_frame,textvariable=_status_msg,anchor=W)
_status.grid(row=0,column=0,sticky=(E,W))
Weendthelayoutsectionbyplacingdownthestatusframe,whichisasimplettk.Frame.
Togiveitalittlestatusbareffect,wesetitsreliefpropertyto'sunken'andgiveita
uniformpaddingof2pixels.Itneedstosticktothe_rootwindowleft,rightandbottom
parts,sowesetitsstickyattributeto(E,W,S).
Wethenplacealabelinitand,thistime,wetieittoaStringVarobject,becausewewill
WOW! eBook
www.wowebook.org
havetomodifyiteverytimewewanttoupdatethestatusbartext.Youshouldbe
acquaintedtothistechniquebynow.
Finally,onthelastline,weruntheapplicationbycallingthemainloopmethodontheTk
instance.
_root.mainloop()
Pleaserememberthatalltheseinstructionsareplacedundertheif__name__==
"__main__":clauseintheoriginalscript.
Asyoucansee,thecodetodesignourGUIapplicationisnothard.Granted,atthe
beginningyouhavetoplayaroundalittlebit.Noteverythingwillworkoutperfectlyat
thefirstattempt,butIpromiseyouit’sveryeasyandyoucanfindplentyoftutorialson
theweb.Let’snowgettotheinterestingbit,thebusinesslogic.
WOW! eBook
www.wowebook.org
Thebusinesslogic
We’llanalyzethebusinesslogicoftheGUIapplicationinthreechunks.Thereisthe
fetchinglogic,thesavinglogic,andthealertinglogic.
Fetchingthewebpage
config={}
deffetch_url():
url=_url.get()
config['images']=[]
_images.set(())#initializedasanemptytuple
try:
page=requests.get(url)
exceptrequests.RequestExceptionasrex:
_sb(str(rex))
else:
soup=BeautifulSoup(page.content,'html.parser')
images=fetch_images(soup,url)
ifimages:
_images.set(tuple(img['name']forimginimages))
_sb('Imagesfound:{}'.format(len(images)))
else:
_sb('Noimagesfound')
config['images']=images
deffetch_images(soup,base_url):
images=[]
forimginsoup.findAll('img'):
src=img.get('src')
img_url=(
'{base_url}/{src}'.format(base_url=base_url,src=src))
name=img_url.split('/')[-1]
images.append(dict(name=name,url=img_url))
returnimages
Firstofall,letmeexplainthatconfigdictionary.Weneedsomewayofpassingdata
betweentheGUIapplicationandthebusinesslogic.Now,insteadofpollutingtheglobal
namespacewithmanydifferentvariables,mypersonalpreferenceistohaveasingle
dictionarythatholdsalltheobjectsweneedtopassbackandforth,sothattheglobal
namespaceisn’tbecloggedupwithallthosenames,andwehaveonesingle,clean,easy
wayofknowingwherealltheobjectsthatareneededbyourapplicationare.
Inthissimpleexample,we’lljustpopulatetheconfigdictionarywiththeimageswefetch
fromthepage,butIwantedtoshowyouthetechniquesothatyouhaveatleastan
example.ThistechniquecomesfrommyexperiencewithJavaScript.Whenyoucodea
webpage,youveryoftenimportseveraldifferentlibraries.Ifeachoftheseclutteredthe
globalnamespacewithallsortsofvariables,therewouldbesevereissuesinmaking
everythingwork,becauseofnameclashesandvariableoverriding.Theymakethecoder’s
lifealivinghell.
So,it’smuchbettertotryandleavetheglobalnamespaceascleanaswecan.Inthiscase,
WOW! eBook
www.wowebook.org
Ifindthatusingoneconfigvariableismorethanacceptable.
Thefetch_urlfunctionisquitesimilartowhatwedidinthescript.Firstly,wegetthe
urlvaluebycalling_url.get().Rememberthatthe_urlobjectisaStringVarinstance
thatistiedtothe_url_entryobject,whichisanEntry.ThetextfieldyouseeontheGUI
istheEntry,butthetextbehindthescenesisthevalueoftheStringVarobject.
Bycallingget()on_url,wegetthevalueofthetextwhichisdisplayedin_url_entry.
Thenextstepistoprepareconfig['images']tobeanemptylist,andtoemptythe
_imagesvariable,whichistiedto_img_listbox.This,ofcourse,hastheeffectof
cleaningupalltheitemsin_img_listbox.
Afterthispreparationstep,wecantrytofetchthepage,usingthesametry/exceptlogic
weadoptedinthescriptatthebeginningofthechapter.
Theonedifferenceisintheactionwetakeifthingsgowrong.Wecall_sb(str(rex)).
_sbisahelperfunctionwhosecodewe’llseeshortly.Basically,itsetsthetextinthestatus
barforus.Notagoodname,right?Ihadtoexplainitsbehaviortoyou:foodforthought.
Ifwecanfetchthepage,thenwecreatethesoupinstance,andfetchtheimagesfromit.
Thelogicoffetch_imagesisexactlythesameastheoneexplainedbefore,soIwon’t
repeatmyselfhere.
Ifwehaveimages,usingaquicktuplecomprehension(whichisactuallyagenerator
expressionfedtoatupleconstructor)wefeedthe_imagesStringVarandthishasthe
effectofpopulatingour_img_listboxwithalltheimagenames.Finally,weupdatethe
statusbar.
Iftherewerenoimages,westillupdatethestatusbar,andattheendofthefunction,
regardlessofhowmanyimageswerefound,weupdateconfig['images']toholdthe
imageslist.Inthisway,we’llbeabletoaccesstheimagesfromotherfunctionsby
inspectingconfig['images']withouthavingtopassthatlistaround.
Savingtheimages
Thelogictosavetheimagesisprettystraightforward.Hereitis:
defsave():
ifnotconfig.get('images'):
_alert('Noimagestosave')
return
if_save_method.get()=='img':
dirname=filedialog.askdirectory(mustexist=True)
_save_images(dirname)
else:
filename=filedialog.asksaveasfilename(
initialfile='images.json',
filetypes=[('JSON','.json')])
_save_json(filename)
def_save_images(dirname):
ifdirnameandconfig.get('images'):
WOW! eBook
www.wowebook.org
forimginconfig['images']:
img_data=requests.get(img['url']).content
filename=os.path.join(dirname,img['name'])
withopen(filename,'wb')asf:
f.write(img_data)
_alert('Done')
def_save_json(filename):
iffilenameandconfig.get('images'):
data={}
forimginconfig['images']:
img_data=requests.get(img['url']).content
b64_img_data=base64.b64encode(img_data)
str_img_data=b64_img_data.decode('utf-8')
data[img['name']]=str_img_data
withopen(filename,'w')asijson:
ijson.write(json.dumps(data))
_alert('Done')
WhentheuserclickstheScrapebutton,thesavefunctioniscalledusingthecallback
mechanism.
Thefirstthingthatthisfunctiondoesischeckwhetherthereareactuallyanyimagestobe
saved.Ifnot,italertstheuseraboutit,usinganotherhelperfunction,_alert,whosecode
we’llseeshortly.Nofurtheractionisperformediftherearenoimages.
Ontheotherhand,iftheconfig['images']listisnotempty,saveactsasadispatcher,
anditcalls_save_imagesor_save_json,accordingtowhichvalueisheldby
_same_method.Remember,thisvariableistiedtotheradiobuttons,thereforeweexpectits
valuetobeeither'img'or'json'.
Thisdispatcherisabitdifferentfromtheoneinthescript.Accordingtowhichmethodwe
haveselected,adifferentactionmustbetaken.
Ifwewanttosavetheimagesasimages,weneedtoasktheusertochooseadirectory.We
dothisbycallingfiledialog.askdirectoryandassigningtheresultofthecalltothe
variabledirname.Thisopensupanicedialogwindowthatasksustochooseadirectory.
Thedirectorywechoosemustexist,asspecifiedbythewaywecallthemethod.Thisis
donesothatwedon’thavetowritecodetodealwithapotentiallymissingdirectorywhen
savingthefiles.
Here’showthisdialogshouldlookonUbuntu:
WOW! eBook
www.wowebook.org
Ifwecanceltheoperation,dirnamewillbesettoNone.
Beforefinishinganalyzingthelogicinsave,let’squicklygothrough_save_images.
It’sverysimilartotheversionwehadinthescriptsojustnotethat,atthebeginning,in
ordertobesurethatweactuallyhavesomethingtodo,wecheckonbothdirnameandthe
presenceofatleastoneimageinconfig['images'].
Ifthat’sthecase,itmeanswehaveatleastoneimagetosaveandthepathforit,sowecan
proceed.Thelogictosavetheimageshasalreadybeenexplained.Theonethingwedo
differentlythistimeistojointhedirectory(whichmeansthecompletepath)totheimage
name,bymeansofos.path.join.Intheos.pathmodulethere’splentyofusefulmethods
toworkwithpathsandfilenames.
Attheendof_save_images,ifwesavedatleastoneimage,wealerttheuserthatwe’re
done.
Let’sgobacknowtotheotherbranchinsave.Thisbranchisexecutedwhentheuser
selectstheAsJSONradiobuttonbeforepressingtheScrapebutton.Inthiscase,wewant
tosaveafile;therefore,wecannotjustaskforadirectory.Wewanttogivetheuserthe
abilitytochooseafilenameaswell.Hence,wefireupadifferentdialog:
filedialog.asksaveasfilename.
Wepassaninitialfilename,whichisproposedtotheuserwiththeabilitytochangeitif
theydon’tlikeit.Moreover,becausewe’resavingaJSONfile,we’reforcingtheuserto
WOW! eBook
www.wowebook.org
usethecorrectextensionbypassingthefiletypesargument.Itisalistwithanynumber
of2-tuples(description,extension)thatrunsthelogicofthedialog.
Here’showthisdialogshouldlookonUbuntu:
Oncewehavechosenaplaceandafilename,wecanproceedwiththesavinglogic,which
isthesameasitwasinthepreviousscript.WecreateaJSONobjectfromaPython
dictionary(data)thatwepopulatewithkey/valuepairsmadebytheimagesnameand
Base64encodedcontent.
In_save_jsonaswell,wehavealittlecheckatthebeginningthatmakessurethatwe
don’tproceedunlesswehaveafilenameandatleastoneimagetosave.
ThisensuresthatiftheuserpressestheCancelbutton,nothingbadhappens.
Alertingtheuser
Finally,let’sseethealertinglogic.It’sextremelysimple.
def_sb(msg):
_status_msg.set(msg)
def_alert(msg):
messagebox.showinfo(message=msg)
That’sit!Tochangethestatusbarmessageallweneedtodoistoaccess_status_msg
StringVar,asit’stiedtothe_statuslabel.
WOW! eBook
www.wowebook.org
Ontheotherhand,ifwewanttoshowtheuseramorevisiblemessage,wecanfireupa
messagebox.Here’showitshouldlookonUbuntu:
Themessageboxobjectcanalsobeusedtowarntheuser(messagebox.showwarning)orto
signalanerror(messagebox.showerror).Butitcanalsobeusedtoprovidedialogsthat
askusifwe’resurethatwewanttoproceedorifwereallywanttodeletethatfile,andso
on.
Ifyouinspectmessageboxbysimplyprintingoutwhatdir(messagebox)returns,you’ll
findmethodslikeaskokcancel,askquestion,askretrycancel,askyesno,and
askyesnocancel,aswellasasetofconstantstoverifytheresponseoftheuser,suchas
CANCEL,NO,OK,OKCANCEL,YES,YESNOCANCEL,andsoon.Youcancomparethesetothe
user’schoicesothatyouknowwhatthenextactiontoexecutewhenthedialogcloses.
WOW! eBook
www.wowebook.org
Howtoimprovetheapplication?
Nowthatyou’reaccustomedtothefundamentalsofdesigningaGUIapplication,I’dlike
togiveyousomesuggestionsonhowtomakeoursbetter.
Wecanstartfromthecodequality.Doyouthinkthiscodeisgoodenough,orwouldyou
improveit?Ifso,how?Iwouldtestit,andmakesureit’srobustandcatersforallthe
variousscenariosthatausermightcreatebyclickingaroundontheapplication.Iwould
alsomakesurethebehavioriswhatIwouldexpectwhenthewebsitewe’rescrapingis
downforanyreason.
Anotherthingthatwecouldimproveisthenaming.Ihaveprudentlynamedallthe
componentswithaleadingunderscore,bothtohighlighttheirsomewhat“private”nature,
andtoavoidhavingnameclasheswiththeunderlyingobjectstheyarelinkedto.Butin
retrospect,manyofthosecomponentscoulduseabettername,soit’sreallyuptoyouto
refactoruntilyoufindtheformthatsuitsyoubest.Youcouldstartbygivingabettername
tothe_sbfunction!
Forwhatconcernstheuserinterface,youcouldtryandresizethemainapplication.See
whathappens?Thewholecontentstaysexactlywhereitis.Emptyspaceisaddedifyou
expand,orthewholewidgetssetdisappearsgraduallyifyoushrink.Thisbehaviorisn’t
exactlynice,thereforeonequicksolutioncouldbetomaketherootwindowfixed(thatis,
unabletoresize).
Anotherthingthatyoucoulddotoimprovetheapplicationistoaddthesamefunctionality
wehadinthescript,tosaveonlyPNGsorJPGs.Inordertodothis,youcouldplacea
comboboxsomewhere,withthreevalues:All,PNGs,JPGs,orsomethingsimilar.The
usershouldbeabletoselectoneofthoseoptionsbeforesavingthefiles.
Evenbetter,youcouldchangethedeclarationofListboxsothatit’spossibletoselect
multipleimagesatthesametime,andonlytheselectedoneswillbesaved.Ifyoumanage
todothis(it’snotashardasitseems,believeme),thenyoushouldconsiderpresentingthe
Listboxabitbetter,maybeprovidingalternatingbackgroundcolorsfortherows.
Anothernicethingyoucouldaddisabuttonthatopensupadialogtoselectafile.Thefile
mustbeoneoftheJSONfilestheapplicationcanproduce.Onceselected,youcouldrun
somelogictoreconstructtheimagesfromtheirBase64-encodedversion.Thelogictodo
thisisverysimple,sohere’sanexample:
withopen('images.json','r')asf:
data=json.loads(f.read())
for(name,b64val)indata.items():
withopen(name,'wb')asf:
f.write(base64.b64decode(b64val))
Asyoucansee,weneedtoopenimages.jsoninreadmode,andgrabthedatadictionary.
Oncewehaveit,wecanloopthroughitsitems,andsaveeachimagewiththeBase64
decodedcontent.I’llleaveituptoyoutotiethislogictoabuttonintheapplication.
WOW! eBook
www.wowebook.org
Anothercoolfeaturethatyoucouldaddistheabilitytoopenupapreviewpanethat
showsanyimageyouselectfromtheListbox,sothattheusercantakeapeekatthe
imagesbeforedecidingtosavethem.
Finally,onelastsuggestionforthisapplicationistoaddamenu.Maybeevenasimple
menuwithFileand?toprovidetheusualHelporAbout.Justforfun.Addingmenusis
notthatcomplicated;youcanaddtext,keyboardshortcuts,images,andsoon.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Wheredowegofromhere?
IfyouareinterestedindiggingdeeperintotheworldofGUIs,thenI’dliketoofferyou
thefollowingsuggestions.
WOW! eBook
www.wowebook.org
Thetkinter.tixmodule
Exploringtkinteranditsthemedwidgetset,tkinter.ttk,willtakeyousometime.
There’smuchtolearnandplaywith.Anotherinterestingmoduletoexplore,whenyou’ll
befamiliarwiththistechnology,istkinter.tix.
Thetkinter.tix(TkInterfaceExtension)moduleprovidesanadditionalveryrichset
ofwidgets.TheneedforthemstemsfromthefactthatthewidgetsinthestandardTk
libraryarefarfromcomplete.
Thetkinter.tixlibraryallowsustosolvethisproblembyprovidingwidgetslikeHList,
ComboBox,Control(orSpinBox),andvariousscrollablewidgets.Altogether,thereare
morethan40widgets.Theyallowyoutointroducedifferentinteractiontechniquesand
paradigmsintoyourapplications,thusimprovingtheirqualityandusability.
WOW! eBook
www.wowebook.org
Theturtlemodule
Theturtlemoduleisanextendedreimplementationoftheeponymousmodulefromthe
PythonstandarddistributionuptoversionPython2.5.It’saverypopularwaytointroduce
childrentoprogramming.
It’sbasedontheideaofanimaginaryturtlestartingat(0,0)intheCartesianplane.You
canprogrammaticallycommandtheturtletomoveforwardandbackwards,rotate,andso
on.andbycombiningtogetherallthepossiblemoves,allsortsofintricateshapesand
imagescanbedrawn.
It’sdefinitelyworthcheckingout,ifonlytoseesomethingdifferent.
WOW! eBook
www.wowebook.org
wxPython,PyQt,andPyGTK
Afteryouhaveexploredthevastnessofthetkinterrealm,I’dsuggestyoutoexplore
otherGUIlibraries:wxPython,PyQt,andPyGTK.Youmayfindoutoneoftheseworks
betterforyou,oritmakeseasierforyoutocodetheapplicationyouneed.
Ibelievethatcoderscanrealizetheirideasonlywhentheyareconsciousaboutwhattools
theyhaveavailable.Ifyourtoolsetistoonarrow,yourideasmayseemimpossibleor
extremelyhardtorealize,andtheyriskremainingexactlywhattheyare,justideas.
Ofcourse,thetechnologicalspectrumtodayishumongous,soknowingeverythingisnot
possible;therefore,whenyouareabouttolearnanewtechnologyoranewsubject,my
suggestionistogrowyourknowledgebyexploringbreadthfirst.
Investigateseveralthingsnottoodeeply,andthengodeepwiththeoneorthefewthat
lookedmostpromising.Thiswayyou’llbeabletobeproductivewithatleastonetool,
andwhenthetoolnolongerfitsyourneeds,you’llknowwheretodigdeeper,thanksto
yourpreviousexploration.
WOW! eBook
www.wowebook.org
Theprincipleofleastastonishment
Whendesigninganinterface,therearemanydifferentthingstobearinmind.Oneofthem,
whichformeisthemostimportant,isthelaworprincipleofleastastonishment.It
basicallystatesthatifinyourdesignanecessaryfeaturehasahighastonishingfactor,it
maybenecessarytoredesignyourapplication.Togiveyouoneexample,whenyou’re
usedtoworkingwithWindows,wherethebuttonstominimize,maximizeandclosea
windowareonthetop-rightcorner,it’squitehardtoworkonLinux,wheretheyareatthe
top-leftcorner.You’llfindyourselfconstantlygoingtothetop-rightcorneronlyto
discoveroncemorethatthebuttonsareontheotherside.
Ifacertainbuttonhasbecomesoimportantinapplicationsthatit’snowplacedinaprecise
locationbydesigners,pleasedon’tinnovate.Justfollowtheconvention.Userswillonly
becomefrustratedwhentheyhavetowastetimelookingforabuttonthatisnotwhereit’s
supposedtobe.
ThedisregardforthisruleisthereasonwhyIcannotworkwithproductslikeJira.Ittakes
meminutestodosimplethingsthatshouldrequireseconds.
WOW! eBook
www.wowebook.org
Threadingconsiderations
Thistopicisbeyondthescopeofanintroductorybooklikethis,butIdowanttomention
it.Inanutshell,athreadofexecutionisthesmallestsequenceofprogrammed
instructionsthatcanbemanagedindependentlybyascheduler.Thereasonwehavethe
perceptionthatmoderncomputerscandomanythingsatthesametimeisnotonlydueto
thefactthattheyhavemultipleprocessors.Theyalsosubdividetheworkindifferent
threads,whicharethenworkedoninsequence.Iftheirlifecycleissufficientlyshort,
threadscanbeworkedoninonesinglego,buttypically,whathappensisthattheOS
worksonathreadforalittletime,thenswitchestoanotherone,thentoanotherone,then
backtothefirstone,andsoon.Theorderinwhichtheyareworkedondependson
differentfactors.Theendresultisthat,becausecomputersareextremelyfastindoingthis
switching,weperceivemanythingshappeningatthesametime.
IfyouarecodingaGUIapplicationthatneedstoperformalongrunningoperationwhena
buttonisclicked,youwillseethatyourapplicationwillprobablyfreezeuntiltheoperation
hasbeencarriedout.Inordertoavoidthis,andmaintaintheapplication’sresponsiveness,
youmayneedtorunthattime-expensiveoperationinadifferentthreadsothattheOSwill
beabletodedicatealittlebitoftimetotheGUIeverynowandthen,tokeepitresponsive.
Threadsareanadvancedtopic,especiallyinPython.Gainagoodgraspofthe
fundamentalsfirst,andthenhavefunexploringthem!
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,weworkedonaprojecttogether.Wehavewrittenascriptthatscrapesa
verysimplewebpageandacceptsoptionalcommandsthatalteritsbehaviorindoingso.
WealsocodedaGUIapplicationtodothesamethingbyclickingbuttonsinsteadof
typingonaconsole.IhopeyouenjoyedreadingitandfollowingalongasmuchasI
enjoyedwritingit.
Wesawmanydifferentconceptslikecontextmanagers,workingwithfiles,performing
HTTPrequests,andwe’vetalkedaboutguidelinesforusabilityanddesign.
Ihaveonlybeenabletoscratchthesurface,buthopefully,youhaveagoodstartingpoint
fromwhichtoexpandyourexploration.
Throughoutthechapter,Ihavepointedyouinseveraldifferentwaysonhowtoimprove
theapplication,andIhavechallengedyouwithafewexercisesandquestions.Ihopeyou
havetakenthetimetoplaywiththoseideas.Onecanlearnalotjustbyplayingaround
withfunapplicationsliketheonewe’vecodedtogether.
Inthenextchapter,we’regoingtotalkaboutdatascience,oratleastaboutthetoolsthata
Pythonprogrammerhaswhenitcomestofacingthissubject.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter9.DataScience
“Ifwehavedata,let’slookatdata.Ifallwehaveareopinions,let’sgowithmine.” —JimBarksdale,formerNetscapeCEO
Datascienceisaverybroadterm,andcanassumeseveraldifferentmeaningsaccording
tocontext,understanding,tools,andsoon.Therearecountlessbooksaboutthissubject,
whichisnotsuitableforthefaint-hearted.
Inordertodoproperdatascience,youneedtoknowmathematicsandstatisticsatthevery
least.Then,youmaywanttodigintoothersubjectssuchaspatternrecognitionand
machinelearningand,ofcourse,thereisaplethoraoflanguagesandtoolsyoucanchoose
from.
UnlessItransformintoTheAmazingFabriziointhenextfewminutes,Iwon’tbeableto
talkabouteverything;Iwon’tevengetclosetoit.Therefore,inordertorenderthis
chaptermeaningful,we’regoingtoworkonacoolprojecttogether.
About3yearsago,Iwasworkingforatop-tiersocialmediacompanyinLondon.Istayed
therefor2years,andIwasprivilegedtoworkwithseveralpeoplewhosebrillianceIcan
onlystarttodescribe.WewerethefirstintheworldtohaveaccesstotheTwitterAdsAPI,
andwewerepartnerswithFacebookaswell.Thatmeansalotofdata.
Ouranalystsweredealingwithahugenumberofcampaignsandtheywerestruggling
withtheamountofworktheyhadtodo,sothedevelopmentteamIwasapartoftriedto
helpbyintroducingthemtoPythonandtothetoolsPythongivesyoutodealwithdata.It
wasaveryinterestingjourneythatledmetomentorseveralpeopleinthecompanyand
eventuallytoManilawhere,for2weeks,IgaveintensivetraininginPythonanddata
sciencetoouranalyststhere.
Theprojectwe’regoingtodotogetherinthischapterisalightweightversionofthefinal
exampleIpresentedtomyManilastudents.Ihaverewrittenittoasizethatwillfitthis
chapter,andmadeafewadjustmentshereandthereforteachingpurposes,butallthemain
conceptsarethere,soitshouldbefunandinstructionalforyoutocodealong.
Onourjourney,we’regoingtomeetafewofthetoolsyoucanfindinthePython
ecosystemwhenitcomestodealingwithdata,solet’sstartbytalkingaboutRomangods.
WOW! eBook
www.wowebook.org
IPythonandJupyternotebook
In2001,FernandoPerezwasagraduatestudentinphysicsatCUBoulder,andwastrying
toimprovethePythonshellsothathecouldhavesomenicetieslikethosehewasusedto
whenhewasworkingwithtoolssuchasMathematicaandMaple.Theresultofthateffort
tookthenameIPython.
Inanutshell,thatsmallscriptbeganasanenhancedversionofthePythonshelland,
throughtheeffortofothercodersandeventuallyproperfundingfromseveraldifferent
companies,itbecamethewonderfulandsuccessfulprojectitistoday.Some10yearsafter
itsbirth,anotebookenvironmentwascreated,poweredbytechnologieslikeWebSockets,
theTornadowebserver,jQuery,CodeMirror,andMathJax.TheZeroMQlibrarywasalso
usedtohandlethemessagesbetweenthenotebookinterfaceandthePythoncorethatlies
behindit.
TheIPythonnotebookhasbecomesopopularandwidelyusedthateventually,allsortsof
goodieshavebeenaddedtoit.Itcanhandlewidgets,parallelcomputing,allsortsofmedia
formats,andmuch,muchmore.Moreover,atsomepoint,itbecamepossibletocodein
languagesotherthanPythonfromwithinthenotebook.
Thishasledtoahugeprojectthatonlyrecentlyhasbeensplitintotwo:IPythonhasbeen
strippeddowntofocusmoreonthekernelpartandtheshell,whilethenotebookhas
becomeabrandnewprojectcalledJupyter.Jupyterallowsinteractivescientific
computationstobedoneinmorethan40languages.
Thischapter’sprojectwillallbecodedandruninaJupyternotebook,soletmeexplainin
afewwordswhatanotebookis.
Anotebookenvironmentisawebpagethatexposesasimplemenuandthecellsinwhich
youcanrunPythoncode.Eventhoughthecellsareseparateentitiesthatyoucanrun
individually,theyallsharethesamePythonkernel.Thismeansthatallthenamesthatyou
defineinacell(thevariables,functions,andsoon)willbeavailableinanyothercell.
Note
Simplyput,aPythonkernelisaprocessinwhichPythonisrunning.Thenotebookweb
pageisthereforeaninterfaceexposedtotheuserfordrivingthiskernel.Thewebpage
communicatestoitusingaveryfastmessagingsystem.
Apartfromallthegraphicaladvantages,thebeautytohavesuchanenvironmentconsists
intheabilityofrunningaPythonscriptinchunks,andthiscanbeatremendous
advantage.Takeascriptthatisconnectingtoadatabasetofetchdataandthenmanipulate
thatdata.Ifyoudoitintheconventionalway,withaPythonscript,youhavetofetchthe
dataeverytimeyouwanttoexperimentwithit.Withinanotebookenvironment,youcan
fetchthedatainacellandthenmanipulateandexperimentwithitinothercells,so
fetchingiteverytimeisnotnecessary.
Thenotebookenvironmentisalsoextremelyhelpfulfordatasciencebecauseitallowsfor
step-by-stepintrospection.Youdoonechunkofworkandthenverifyit.Youthendo
WOW! eBook
www.wowebook.org
anotherchunkandverifyagain,andsoon.
It’salsoinvaluableforprototypingbecausetheresultsarethere,rightinfrontofyour
eyes,immediatelyavailable.
Ifyouwanttoknowmoreaboutthesetools,pleasecheckouthttp://ipython.org/and
http://jupyter.org/.
Ihavecreatedaverysimpleexamplenotebookwithafibonaccifunctionthatgivesyou
thelistofallFibonaccinumberssmallerthanagivenN.Inmybrowser,itlookslikethis:
EverycellhasanIn[]label.Ifthere’snothingbetweenthebraces,itmeansthatcellhas
neverbeenexecuted.Ifthereisanumber,itmeansthatthecellhasbeenexecuted,andthe
numberrepresentstheorderinwhichthecellwasexecuted.Finally,a*meansthatthe
celliscurrentlybeingexecuted.
YoucanseeinthepicturethatinthefirstcellIhavedefinedthefibonaccifunction,andI
haveexecutedit.Thishastheeffectofplacingthefibonaccinameintheglobalframe
associatedwiththenotebook,thereforethefibonaccifunctionisnowavailabletothe
othercellsaswell.Infact,inthesecondcell,Icanrunfibonacci(100)andseetheresults
inOut[2].Inthethirdcell,Ihaveshownyouoneoftheseveralmagicfunctionsyoucan
findinanotebookinthesecondcell.%timeitrunsthecodeseveraltimesandprovides
youwithanicebenchmarkforit.Allthemeasurementsforthelistcomprehensionsand
generatorsIdidinChapter5,SavingTimeandMemorywerecarriedoutwiththisnice
feature.
Youcanexecuteacellasmanytimesasyouwant,andchangetheorderinwhichyourun
them.Cellsareverymalleable,youcanalsoputinmarkdowntextorrenderthemas
WOW! eBook
www.wowebook.org
headers.
Note
Markdownisalightweightmarkuplanguagewithplaintextformattingsyntaxdesigned
sothatitcanbeconvertedtoHTMLandmanyotherformats.
Also,whateveryouplaceinthelastrowofacellwillbeautomaticallyprintedforyou.
Thisisveryhandybecauseyou’renotforcedtowriteprint(...)explicitly.
Feelfreetoexplorethenotebookenvironment;onceyou’refriendswithit,it’salonglastingrelationship,Ipromise.
Inordertorunthenotebook,youhavetoinstallahandfuloflibraries,eachofwhich
collaborateswiththeotherstomakethewholethingwork.Alternatively,youcanjust
installJupyteranditwilltakecareofeverythingforyou.Forthischapter,thereareafew
otherdependenciesthatweneedtoinstall,sopleaserunthefollowingcommand:
$pipinstalljupyterpandasmatplotlibfake-factorydeloreanxlwt
Don’tworry,I’llintroduceyoutoeachofthesegradually.Now,whenyou’redone
installingtheselibraries(itmaytakeafewminutes),youcanstartthenotebook:
$jupyternotebook
Thiswillopenapageinyourbrowseratthisaddress:http://localhost:8888/.
Gotothatpageandcreateanewnotebookusingthemenu.Whenyouhaveitandyou’re
comfortablewithit,we’rereadytogo.
Tip
Ifyouexperienceanyissuessettingupthenotebookenvironment,pleasedon’tget
discouraged.Ifyougetanerror,it’susuallyjustamatterofsearchingalittlebitonthe
webandyou’llenduponapagewheresomeoneelsehashadthesameissue,andthey
haveexplainedhowtofixit.Tryyourbesttohavethenotebookenvironmentupand
runningbeforecontinuingwiththechapter.
Ourprojectwilltakeplaceinanotebook,thereforeIwilltageachcodesnippetwiththe
cellnumberitbelongsto,sothatyoucaneasilyreproducethecodeandfollowalong.
Tip
Ifyoufamiliarizeyourselfwiththekeyboardshortcuts(lookinthenotebook’shelp
section),youwillbeabletomovebetweencellsandhandletheircontentwithouthaving
toreachforthemouse.Thiswillmakeyoumoreproficientandwayfasterwhenyouwork
inanotebook.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Dealingwithdata
Typically,whenyoudealwithdata,thisisthepathyougothrough:youfetchit,youclean
andmanipulateit,thenyouinspectitandpresentresultsasvalues,spreadsheets,graphs,
andsoon.Iwantyoutobeinchargeofallthreestepsoftheprocesswithouthavingany
externaldependencyonadataprovider,sowe’regoingtodothefollowing:
1. We’regoingtocreatethedata,simulatingthefactthatitcomesinaformatwhichis
notperfectorreadytobeworkedon.
2. We’regoingtocleanitandfeedittothemaintoolwe’lluseintheproject:
DataFrameofpandas.
3. We’regoingtomanipulatethedataintheDataFrame.
4. We’regoingtosavetheDataFrametoafileindifferentformats.
5. Finally,we’regoingtoinspectthedataandgetsomeresultsoutofit.
WOW! eBook
www.wowebook.org
Settingupthenotebook
Firstthingsfirst,weneedtosetupthenotebook.Thismeansimportsandabitof
configuration.
#1
importjson
importcalendar
importrandom
fromdatetimeimportdate,timedelta
importfaker
importnumpyasnp
frompandasimportDataFrame
fromdeloreanimportparse
importpandasaspd
#makethegraphsnicer
pd.set_option('display.mpl_style','default')
Cell#1takescareoftheimports.Therearequiteafewnewthingshere:thecalendar,
randomanddatetimemodulesarepartofthestandardlibrary.Theirnamesareselfexplanatory,solet’slookatfaker.Thefake-factorylibrarygivesyouthismodule,
whichyoucanusetopreparefakedata.It’sveryusefulintests,whenyouprepareyour
fixtures,togetallsortsofthingssuchasnames,e-mailaddresses,phonenumbers,credit
carddetails,andmuchmore.Itisallfake,ofcourse.
numpyistheNumPylibrary,thefundamentalpackageforscientificcomputingwith
Python.I’llspendafewwordsonitlateroninthechapter.
pandasistheverycoreuponwhichthewholeprojectisbased.ItstandsforPythonData
AnalysisLibrary.Amongmanyothers,itprovidestheDataFrame,amatrix-likedata
structurewithadvancedprocessingcapabilities.It’scustomarytoimporttheDataFrame
separatelyandthendoimportpandasaspd.
deloreanisanicethird-partylibrarythatspeedsupdealingwithdatesdramatically.
Technically,wecoulddoitwiththestandardlibrary,butIseenoreasonnottoexpanda
bittherangeoftheexampleandshowyousomethingdifferent.
Finally,wehaveaninstructiononthelastlinethatwillmakeourgraphsattheendalittle
bitnicer,whichdoesn’thurt.
WOW! eBook
www.wowebook.org
Preparingthedata
Wewanttoachievethefollowingdatastructure:we’regoingtohavealistofuserobjects.
Eachuserobjectwillbelinkedtoanumberofcampaignobjects.
InPython,everythingisanobject,soI’musingthisterminagenericway.Theuserobject
maybeastring,adict,orsomethingelse.
Acampaigninthesocialmediaworldisapromotionalcampaignthatamediaagency
runsonsocialmedianetworksonbehalfofaclient.
Rememberthatwe’regoingtopreparethisdatasothatit’snotinperfectshape(butit
won’tbesobadeither…).
#2
fake=faker.Faker()
Firstly,weinstantiatetheFakerthatwe’llusetocreatethedata.
#3
usernames=set()
usernames_no=1000
#populatethesetwith1000uniqueusernames
whilelen(usernames)<usernames_no:
usernames.add(fake.user_name())
Thenweneedusernames.Iwant1,000uniqueusernames,soIloopoverthelengthofthe
usernamessetuntilithas1,000elements.Asetdoesn’tallowduplicatedelements,
thereforeuniquenessisguaranteed.
#4
defget_random_name_and_gender():
skew=.6#60%ofuserswillbefemale
male=random.random()>skew
ifmale:
returnfake.name_male(),'M'
else:
returnfake.name_female(),'F'
defget_users(usernames):
users=[]
forusernameinusernames:
name,gender=get_random_name_and_gender()
user={
'username':username,
'name':name,
'gender':gender,
'email':fake.email(),
'age':fake.random_int(min=18,max=90),
'address':fake.address(),
}
users.append(json.dumps(user))
returnusers
WOW! eBook
www.wowebook.org
users=get_users(usernames)
users[:3]
Here,wecreatealistofusers.Eachusernamehasnowbeenaugmentedtoafull-blown
userdict,withotherdetailssuchasname,gender,e-mail,andsoon.Eachuserdictis
thendumpedtoJSONandaddedtothelist.Thisdatastructureisnotoptimal,ofcourse,
butwe’resimulatingascenariowhereuserscometouslikethat.
Notetheskeweduseofrandom.random()tomake60%ofusersfemale.Therestofthe
logicshouldbeveryeasyforyoutounderstand.
Notealsothelastline.Eachcellautomaticallyprintswhat’sonthelastline;therefore,the
outputofthisisalistwiththefirstthreeusers:
Out#4
['{"gender":"F","age":48,"email":"[email protected]",
"address":"2006SawaynTrailApt.207\\nHyattview,MO27278","username":
"darcy00","name":"VirgiaHilpert"}',
'{"gender":"F","age":58,"email":"[email protected]","address":
"5176AndresPlainsApt.040\\nLakinside,GA92446","username":
"renner.virgie","name":"MissClarabelleKertzmannMD"}',
'{"gender":"M","age":33,"email":"[email protected]",
"address":"1218JacobsonFort\\nNorthDoctor,OK04469","username":
"hettinger.alphonsus","name":"LudwigProsacco"}']
Note
Ihopeyou’refollowingalongwithyourownnotebook.Ifyoudo,pleasenotethatalldata
isgeneratedusingrandomfunctionsandvalues;therefore,youwillseedifferentresults.
Theywillchangeeverytimeyouexecutethenotebook.
#5
#campaignnameformat:
#InternalType_StartDate_EndDate_TargetAge_TargetGender_Currency
defget_type():
#justsomegibberishinternalcodes
types=['AKX','BYU','GRZ','KTR']
returnrandom.choice(types)
defget_start_end_dates():
duration=random.randint(1,2*365)
offset=random.randint(-365,365)
start=date.today()-timedelta(days=offset)
end=start+timedelta(days=duration)
def_format_date(date_):
returndate_.strftime("%Y%m%d")
return_format_date(start),_format_date(end)
defget_age():
age=random.randint(20,45)
age-=age%5
WOW! eBook
www.wowebook.org
diff=random.randint(5,25)
diff-=diff%5
return'{}-{}'.format(age,age+diff)
defget_gender():
returnrandom.choice(('M','F','B'))
defget_currency():
returnrandom.choice(('GBP','EUR','USD'))
defget_campaign_name():
separator='_'
type_=get_type()
start_end=separator.join(get_start_end_dates())
age=get_age()
gender=get_gender()
currency=get_currency()
returnseparator.join(
(type_,start_end,age,gender,currency))
In#5,wedefinethelogictogenerateacampaignname.Analystsusespreadsheetsallthe
timeandtheycomeupwithallsortsofcodingtechniquestocompressasmuch
informationaspossibleintothecampaignnames.TheformatIchoseisasimpleexample
ofthattechnique:thereisacodethattellsthecampaigntype,thenstartandenddates,then
thetargetageandgender,andfinallythecurrency.Allvaluesareseparatedbyan
underscore.
Intheget_typefunction,Iuserandom.choice()togetonevaluerandomlyoutofa
collection.Probablymoreinterestingisget_start_end_dates.First,Igetthedurationfor
thecampaign,whichgoesfrom1dayto2years(randomly),thenIgetarandomoffsetin
timewhichIsubtractfromtoday’sdateinordertogetthestartdate.Giventhattheoffset
isarandomnumberbetween-365and365,wouldanythingbedifferentifIaddeditto
today’sdateinsteadofsubtractingit?
WhenIhaveboththestartandenddates,Ireturnastringifiedversionofthem,joinedby
anunderscore.
Then,wehaveabitofmodulartrickerygoingonwiththeagecalculation.Ihopeyou
rememberthemodulooperator(%)fromChapter2,Built-inDataTypes.
WhathappenshereisthatIwantadaterangethathasmultiplesof5asextremes.So,there
aremanywaystodoit,butwhatIdoistogetarandomnumberbetween20and45forthe
leftextreme,andremovetheremainderofthedivisionby5.So,if,forexample,Iget28,I
willremove28%5=3toit,getting25.Icouldhavejustusedrandom.randrange(),but
it’shardtoresistmodulardivision.
Therestofthefunctionsarejustsomeotherapplicationsofrandom.choice()andthelast
one,get_campaign_name,isnothingmorethanacollectorforallthesepuzzlepiecesthat
returnsthefinalcampaignname.
#6
defget_campaign_data():
WOW! eBook
www.wowebook.org
name=get_campaign_name()
budget=random.randint(10**3,10**6)
spent=random.randint(10**2,budget)
clicks=int(random.triangular(10**2,10**5,0.2*10**5))
impressions=int(random.gauss(0.5*10**6,2))
return{
'cmp_name':name,
'cmp_bgt':budget,
'cmp_spent':spent,
'cmp_clicks':clicks,
'cmp_impr':impressions
}
In#6,wewriteafunctionthatcreatesacompletecampaignobject.Iusedafewdifferent
functionsfromtherandommodule.random.randint()givesyouanintegerbetweentwo
extremes.Theproblemwithitisthatitfollowsauniformprobabilitydistribution,which
meansthatanynumberintheintervalhasthesameprobabilityofcomingup.
Therefore,whendealingwithalotofdata,ifyoudistributeyourfixturesusingauniform
distribution,theresultsyouwillgetwillalllooksimilar.Forthisreason,Ichosetouse
triangularandgauss,forclicksandimpressions.Theyusedifferentprobability
distributionssothatwe’llhavesomethingmoreinterestingtoseeintheend.
Justtomakesurewe’reonthesamepagewiththeterminology:clicksrepresentsthe
numberofclicksonacampaignadvertisement,budgetisthetotalamountofmoney
allocatedforthecampaign,spentishowmuchofthatmoneyhasalreadybeenspent,and
impressionsisthenumberoftimesthecampaignhasbeenfetched,asaresource,fromits
source,regardlessoftheamountofclicksthatwereperformedonthecampaign.
Normally,theamountofimpressionsisgreaterthantheamountofclicks.
Nowthatwehavethedata,it’stimetoputitalltogether:
#7
defget_data(users):
data=[]
foruserinusers:
campaigns=[get_campaign_data()
for_inrange(random.randint(2,8))]
data.append({'user':user,'campaigns':campaigns})
returndata
Asyoucansee,eachitemindataisadictwithauserandalistofcampaignsthatare
associatedwiththatuser.
WOW! eBook
www.wowebook.org
Cleaningthedata
Let’sstartcleaningthedata:
#8
rough_data=get_data(users)
rough_data[:2]#let'stakeapeek
Wesimulatefetchingthedatafromasourceandtheninspectit.Thenotebookisthe
perfecttooltoinspectyoursteps.Youcanvarythegranularitytoyourneeds.Thefirst
iteminrough_datalookslikethis:
[{'campaigns':[{'cmp_bgt':130532,
'cmp_clicks':25576,
'cmp_impr':500001,
'cmp_name':'AKX_20150826_20170305_35-50_B_EUR',
'cmp_spent':57574},
...omit…
{'cmp_bgt':884396,
'cmp_clicks':10955,
'cmp_impr':499999,
'cmp_name':'KTR_20151227_20151231_45-55_B_GBP',
'cmp_spent':318887}],
'user':'{"age":44,"username":"jacob43",
"name":"HollandStrosin",
"email":"[email protected]",
"address":"1038RunolfsdottirParks\\nElmapo…",
"gender":"M"}'}]
So,wenowstartworkingwithit.
#9
data=[]
fordatuminrough_data:
forcampaignindatum['campaigns']:
campaign.update({'user':datum['user']})
data.append(campaign)
data[:2]#let'stakeanotherpeek
ThefirstthingweneedtodoinordertobeabletofeedaDataFramewiththisdataisto
denormalizeit.Thismeanstransformingthedataintoalistwhoseitemsarecampaign
dicts,augmentedwiththeirrelativeuserdict.Userswillbeduplicatedineachcampaign
theybelongto.Thefirstitemindatalookslikethis:
[{'cmp_bgt':130532,
'cmp_clicks':25576,
'cmp_impr':500001,
'cmp_name':'AKX_20150826_20170305_35-50_B_EUR',
'cmp_spent':57574,
'user':'{"age":44,"username":"jacob43",
"name":"HollandStrosin",
"email":"[email protected]",
"address":"1038RunolfsdottirParks\\nElmaport…",
"gender":"M"}'}]
WOW! eBook
www.wowebook.org
Youcanseethattheuserobjecthasbeenbroughtintothecampaigndictwhichwas
repeatedforeachcampaign.
WOW! eBook
www.wowebook.org
CreatingtheDataFrame
Nowit’stimetocreatetheDataFrame:
#10
df=DataFrame(data)
df.head()
Finally,wewillcreatetheDataFrameandinspectthefirstfiverowsusingthehead
method.Youshouldseesomethinglikethis:
Jupyterrenderstheoutputofthedf.head()callasHTMLautomatically.Inordertohave
atext-basedoutput,simplywrapdf.head()inaprintcall.
TheDataFramestructureisverypowerful.Itallowsustodoagreatdealofmanipulation
onitscontents.Youcanfilterbyrows,columns,aggregateondata,andmanyother
operations.Youcanoperatewithrowsorcolumnswithoutsufferingthetimepenaltyyou
wouldhavetopayifyouwereworkingondatawithpurePython.Thishappensbecause,
underthecovers,pandasisharnessingthepoweroftheNumPylibrary,whichitselfdraws
itsincrediblespeedfromthelow-levelimplementationofitscore.NumPystandsfor
NumericPython,anditisoneofthemostwidelyusedlibrariesinthedatascience
environment.
UsingaDataFrameallowsustocouplethepowerofNumPywithspreadsheet-like
capabilitiessothatwe’llbeabletoworkonourdatainafashionthatissimilartowhatan
analystcoulddo.Only,wedoitwithcode.
Butlet’sgobacktoourproject.Let’sseetwowaystoquicklygetabird’seyeviewofthe
data:
#11
df.count()
countyieldsacountofallthenon-emptycellsineachcolumn.Thisisgoodtohelpyou
understandhowsparseyourdatacanbe.Inourcase,wehavenomissingvalues,sothe
outputis:
cmp_bgt4974
cmp_clicks4974
cmp_impr4974
cmp_name4974
WOW! eBook
www.wowebook.org
cmp_spent4974
user4974
dtype:int64
Nice!Wehave4,974rows,andthedatatypeisintegers(dtype:int64meanslong
integersbecausetheytake64bitseach).Giventhatwehave1,000usersandtheamount
ofcampaignsperuserisarandomnumberbetween2and8,we’reexactlyinlinewith
whatIwasexpecting.
#12
df.describe()
describeisaniceandquickwaytointrospectabitfurther:
cmp_bgtcmp_clickscmp_imprcmp_spent
count4974.0000004974.0000004974.0000004974.000000
mean503272.70687640225.764978499999.495979251150.604343
std289393.74746521910.6319502.035355220347.594377
min1250.000000609.000000499992.000000142.000000
25%253647.50000022720.750000499998.00000067526.750000
50%508341.00000036561.500000500000.000000187833.000000
75%757078.25000055962.750000500001.000000385803.750000
max999631.00000098767.000000500006.000000982716.000000
Asyoucansee,itgivesusseveralmeasuressuchascount,mean,std(standarddeviation),
min,max,andshowshowdataisdistributedinthevariousquadrants.Thankstothis
method,wecouldalreadyhavearoughideaofhowourdataisstructured.
Let’sseewhicharethethreecampaignswiththehighestandlowestbudgets:
#13
df.sort_index(by=['cmp_bgt'],ascending=False).head(3)
Thisgivesthefollowingoutput(truncated):
cmp_bgtcmp_clickscmp_imprcmp_name
465599963115343499997AKX_20160814_20180226_40
370899960645367499997KTR_20150523_20150527_35
199599944512580499998AKX_20141102_20151009_30
And(#14)acallto.tail(3),showsustheoneswiththelowestbudget.
Unpackingthecampaignname
Nowit’stimetoincreasethecomplexityupabit.Firstofall,wewanttogetridofthat
horriblecampaignname(cmp_name).Weneedtoexplodeitintopartsandputeachpartin
onededicatedcolumn.Inordertodothis,we’llusetheapplymethodoftheSeriesobject.
Thepandas.core.series.Seriesclassisbasicallyapowerfulwrapperaroundanarray
(thinkofitasalistwithaugmentedcapabilities).WecanextrapolateaSeriesobjectfrom
aDataFramebyaccessingitinthesamewaywedowithakeyinadict,andwecancall
applyonthatSeriesobject,whichwillrunafunctionfeedingeachitemintheSeriesto
it.WecomposetheresultintoanewDataFrame,andthenjointhatDataFramewithour
beloveddf.
WOW! eBook
www.wowebook.org
#15
defunpack_campaign_name(name):
#veryoptimisticmethod,assumesdataincampaignname
#isalwaysingoodstate
type_,start,end,age,gender,currency=name.split('_')
start=parse(start).date
end=parse(end).date
returntype_,start,end,age,gender,currency
campaign_data=df['cmp_name'].apply(unpack_campaign_name)
campaign_cols=[
'Type','Start','End','Age','Gender','Currency']
campaign_df=DataFrame(
campaign_data.tolist(),columns=campaign_cols,index=df.index)
campaign_df.head(3)
Withinunpack_campaign_name,wesplitthecampaignnameinparts.Weuse
delorean.parse()togetaproperdateobjectoutofthosestrings(deloreanmakesit
reallyeasytodoit,doesn’tit?),andthenwereturntheobjects.Aquickpeekatthelast
linereveals:
TypeStartEndAgeGenderCurrency
0KTR2016-06-162017-01-2420-30MEUR
1BYU2014-10-252015-07-3135-50BUSD
2BYU2015-10-262016-03-1735-50MEUR
Nice!Oneimportantthing:evenifthedatesappearasstrings,theyarejustthe
representationoftherealdateobjectsthatarehostedintheDataFrame.
Anotherveryimportantthing:whenjoiningtwoDataFrameinstances,it’simperativethat
theyhavethesameindex,otherwisepandaswon’tbeabletoknowwhichrowsgowith
which.Therefore,whenwecreatecampaign_df,wesetitsindextotheonefromdf.This
enablesustojointhem.WhencreatingthisDataFrame,wealsopassthecolumnsnames.
#16
df=df.join(campaign_df)
Andafterthejoin,wetakeapeek,hopingtoseematchingdata(outputtruncated):
#17
df[['cmp_name']+campaign_cols].head(3)
Gives:
cmp_nameTypeStartEnd
0KTR_20160616_20170124_20-30_M_EURKTR2016-06-162017-01-24
1BYU_20141025_20150731_35-50_B_USDBYU2014-10-252015-07-31
2BYU_20151026_20160317_35-50_M_EURBYU2015-10-262016-03-17
Asyoucansee,thejoinwassuccessful;thecampaignnameandtheseparatecolumns
exposethesamedata.Didyouseewhatwedidthere?We’reaccessingtheDataFrame
usingthesquarebracketssyntax,andwepassalistofcolumnnames.Thiswillproducea
brandnewDataFrame,withthosecolumns(inthesameorder),onwhichwethencall
WOW! eBook
www.wowebook.org
head().
Unpackingtheuserdata
WenowdotheexactsamethingforeachpieceofuserJSONdata.Wecallapplyonthe
userSeries,runningtheunpack_user_jsonfunction,whichtakesaJSONuserobjectand
transformsitintoalistofitsfields,whichwecantheninjectintoabrandnewDataFrame
user_df.Afterthat,we’lljoinuser_dfbackwithdf,likewedidwithcampaign_df.
#18
defunpack_user_json(user):
#veryoptimisticaswell,expectsuserobjects
#tohaveallattributes
user=json.loads(user.strip())
return[
user['username'],
user['email'],
user['name'],
user['gender'],
user['age'],
user['address'],
]
user_data=df['user'].apply(unpack_user_json)
user_cols=[
'username','email','name','gender','age','address']
user_df=DataFrame(
user_data.tolist(),columns=user_cols,index=df.index)
Verysimilartothepreviousoperation,isn’tit?Weshouldalsonoteherethat,when
creatinguser_df,weneedtoinstructDataFrameaboutthecolumnnamesand,very
important,theindex.Let’sjoin(#19)andtakeaquickpeek(#20):
df=df.join(user_df)
df[['user']+user_cols].head(2)
Theoutputshowsusthateverythingwentwell.We’regood,butwe’renotdoneyet.
Ifyoucalldf.columnsinacell,you’llseethatwestillhaveuglynamesforourcolumns.
Let’schangethat:
#21
better_columns=[
'Budget','Clicks','Impressions',
'cmp_name','Spent','user',
'Type','Start','End',
'TargetAge','TargetGender','Currency',
'Username','Email','Name',
'Gender','Age','Address',
]
df.columns=better_columns
Good!Now,withtheexceptionof'cmp_name'and'user',weonlyhavenicenames.
CompletingthedatasetNextstepwillbetoaddsomeextracolumns.Foreachcampaign,
WOW! eBook
www.wowebook.org
wehavetheamountofclicksandimpressions,andwehavethespent.Thisallowsusto
introducethreemeasurementratios:CTR,CPC,andCPI.TheystandforClickThrough
Rate,CostPerClick,andCostPerImpression,respectively.
Thelasttwoareeasytounderstand,butCTRisnot.Sufficeittosaythatitistheratio
betweenclicksandimpressions.Itgivesyouameasureofhowmanyclickswere
performedonacampaignadvertisementperimpression:thehigherthisnumber,themore
successfultheadvertisementisinattractinguserstoclickonit.
#22
defcalculate_extra_columns(df):
#ClickThroughRate
df['CTR']=df['Clicks']/df['Impressions']
#CostPerClick
df['CPC']=df['Spent']/df['Clicks']
#CostPerImpression
df['CPI']=df['Spent']/df['Impressions']
calculate_extra_columns(df)
Iwrotethisasafunction,butIcouldhavejustwrittenthecodeinthecell.It’snot
important.WhatIwantyoutonoticehereisthatwe’readdingthosethreecolumnswith
onelineofcodeeach,buttheDataFrameappliestheoperationautomatically(thedivision,
inthiscase)toeachpairofcellsfromtheappropriatecolumns.So,eveniftheyare
maskedasthreedivisions,theseareactually4974*3divisions,becausetheyare
performedforeachrow.Pandasdoesalotofworkforus,andalsodoesaverygoodjobin
hidingthecomplexityofit.
Thefunction,calculate_extra_columns,takesaDataFrame,andworksdirectlyonit.
Thismodeofoperationiscalledin-place.Doyourememberhowlist.sort()was
sortingthelist?Itisthesamedeal.
Wecantakealookattheresultsbyfilteringontherelevantcolumnsandcallinghead.
#23
df[['Spent','Clicks','Impressions',
'CTR','CPC','CPI']].head(3)
Thisshowsusthatthecalculationswereperformedcorrectlyoneachrow:
SpentClicksImpressionsCTRCPCCPI
057574255765000010.0511522.2510950.115148
1226319612474999990.1224943.6951850.452639
24354155825000040.0311640.2794250.008708
Now,Iwanttoverifytheaccuracyoftheresultsmanuallyforthefirstrow:
#24
clicks=df['Clicks'][0]
impressions=df['Impressions'][0]
spent=df['Spent'][0]
CTR=df['CTR'][0]
CPC=df['CPC'][0]
CPI=df['CPI'][0]
WOW! eBook
www.wowebook.org
print('CTR:',CTR,clicks/impressions)
print('CPC:',CPC,spent/clicks)
print('CPI:',CPI,spent/impressions)
Ityieldsthefollowingoutput:
CTR:0.05115189769620.0511518976962
CPC:2.251094776352.25109477635
CPI:0.1151477697040.115147769704
Thisisexactlywhatwesawinthepreviousoutput.Ofcourse,Iwouldn’tnormallyneedto
dothis,butIwantedtoshowyouhowcanyouperformcalculationsthisway.Youcan
accessaSeries(acolumn)bypassingitsnametotheDataFrame,insquarebrackets,and
thenyouaccesseachrowbyitsposition,exactlyasyouwouldwitharegularlistortuple.
We’realmostdonewithourDataFrame.Allwearemissingnowisacolumnthattellsus
thedurationofthecampaignandacolumnthattellsuswhichdayoftheweekcorresponds
tothestartdateofeachcampaign.Thisallowsmetoexpandonhowtoplaywithdate
objects.
#25
defget_day_of_the_week(day):
number_to_day=dict(enumerate(calendar.day_name,1))
returnnumber_to_day[day.isoweekday()]
defget_duration(row):
return(row['End']-row['Start']).days
df['DayofWeek']=df['Start'].apply(get_day_of_the_week)
df['Duration']=df.apply(get_duration,axis=1)
Weusedtwodifferenttechniquesherebutfirst,thecode.
get_day_of_the_weektakesadateobject.Ifyoucannotunderstandwhatitdoes,please
takeafewmomentstotryandunderstandforyourselfbeforereadingtheexplanation.Use
theinside-outtechniquelikewe’vedoneafewtimesbefore.
So,asI’msureyouknowbynow,ifyouputcalendar.day_nameinalistcall,youget
['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday',
'Sunday'].Thismeansthat,ifweenumeratecalendar.day_namestartingfrom1,weget
pairssuchas(1,'Monday'),(2,'Tuesday'),andsoon.Ifwefeedthesepairstoadict,
wegetamappingbetweenthedaysoftheweekasnumbers(1,2,3,…)andtheirnames.
Whenthemappingiscreated,inordertogetthenameofaday,wejustneedtoknowits
number.Togetit,wecalldate.isoweekday(),whichtellsuswhichdayoftheweekthat
dateis(asanumber).Youfeedthatintothemappingand,boom!Youhavethenameof
theday.
get_durationisinterestingaswell.First,noticeittakesanentirerow,notjustasingle
value.Whathappensinitsbodyisthatweperformasubtractionbetweenacampaignend
andstartdates.Whenyousubtractdateobjectstheresultisatimedeltaobject,which
representsagivenamountoftime.Wetakethevalueofits.daysproperty.Itisassimple
asthat.
WOW! eBook
www.wowebook.org
Now,wecanintroducethefunpart,theapplicationofthosetwofunctions.
ThefirstapplicationisperformedonaSeriesobject,likewedidbeforefor'user'and
'cmp_name',thereisnothingnewhere.
ThesecondoneisappliedtothewholeDataFrameand,inordertoinstructPandasto
performthatoperationontherows,wepassaxis=1.
Wecanverifytheresultsveryeasily,asshownhere:
#26
df[['Start','End','Duration','DayofWeek']].head(3)
Yields:
StartEndDurationDayofWeek
02015-08-262017-03-05557Wednesday
12014-10-152014-12-1965Wednesday
22015-02-222016-01-14326Sunday
So,wenowknowthatbetweenthe26thofAugust2015andthe5thofMarch2017there
are557days,andthatthe26thofAugust2015wasaWednesday.
Ifyou’rewonderingwhatthepurposeofthisis,I’llprovideanexample.Imaginethatyou
haveacampaignthatistiedtoasportseventthatusuallytakesplaceonaSunday.You
maywanttoinspectyourdataaccordingtothedayssothatyoucancorrelatethemtothe
variousmeasurementsyouhave.We’renotgoingtodoitinthisproject,butitwasuseful
tosee,ifonlyforthedifferentwayofcallingapply()onaDataFrame.
Cleaningeverythingup
Nowthatwehaveeverythingwewant,it’stimetodothefinalcleaning:rememberwestill
havethe'cmp_name'and'user'columns.Thoseareuselessnow,sotheyhavetogo.
Also,IwanttoreorderthecolumnsintheDataFramesothatitismorerelevanttothedata
itnowcontains.Inordertodothis,wejustneedtofilterdfonthecolumnlistwewant.
We’llgetbackabrandnewDataFramethatwecanreassigntodfitself.
#27
final_columns=[
'Type','Start','End','Duration','DayofWeek','Budget',
'Currency','Clicks','Impressions','Spent','CTR','CPC',
'CPI','TargetAge','TargetGender','Username','Email',
'Name','Gender','Age'
]
df=df[final_columns]
Ihavegroupedthecampaigninformationatthebeginning,thenthemeasurements,and
finallytheuserdataattheend.NowourDataFrameiscleanandreadyforustoinspect.
Beforewestartgoingcrazywithgraphs,whatabouttakingasnapshotofourDataFrame
sothatwecaneasilyreconstructitfromafile,ratherthanhavingtoredoallthestepswe
didtogethere.Someanalystsmaywanttohaveitinspreadsheetform,todoadifferent
kindofanalysisthantheonewewanttodo,solet’sseehowtosaveaDataFrametoafile.
WOW! eBook
www.wowebook.org
It’seasierdonethansaid.
WOW! eBook
www.wowebook.org
SavingtheDataFrametoafile
WecansaveaDataFrameinmanydifferentways.Youcantypedf.to_andthenpress
Tabtomakeauto-completionpopup,toseeallthepossibleoptions.
We’regoingtosaveourDataFrameinthreedifferentformats,justforfun:commaseparatedvalues(CSV),JSON,andExcelspreadsheet.
#28
df.to_csv('df.csv')
#29
df.to_json('df.json')
#30
df.to_excel('df.xls')
TheCSVfilelookslikethis(outputtruncated):
Type,Start,End,Duration,DayofWeek,Budget,Currency,Clicks,Impres
0,GRZ,2015-03-15,2015-11-10,240,Sunday,622551,GBP,35018,500002,787
1,AKX,2016-06-19,2016-09-19,92,Sunday,148219,EUR,45185,499997,6588
2,BYU,2014-09-25,2016-07-03,647,Thursday,537760,GBP,55771,500001,3
AndtheJSONonelikethis(again,outputtruncated):
{
"Type":{
"0":"GRZ",
"1":"AKX",
"2":"BYU",
So,it’sextremelyeasytosaveaDataFrameinmanydifferentformats,andthegoodnews
isthattheoppositeisalsotrue:it’sveryeasytoloadaspreadsheetintoaDataFrame.The
programmersbehindPandaswentalongwaytoeaseourtasks,somethingtobegrateful
for.
WOW! eBook
www.wowebook.org
Visualizingtheresults
Finally,thejuicybits.Inthissection,we’regoingtovisualizesomeresults.Fromadata
scienceperspective,I’mnotveryinterestedingoingdeepintoanalysis,especiallybecause
thedataiscompletelyrandom,butnonetheless,thiscodewillgetyoustartedwithgraphs
andotherfeatures.
SomethingIlearnedinmylife—andthismaycomeasasurprisetoyou—isthatlooks
alsocountssoit’sveryimportantthatwhenyoupresentyourresults,youdoyourbestto
makethempretty.
Iwon’ttrytoprovetoyouhowtruthfulthatlaststatementis,butIreallydobelieveinit.
Ifyourecallthelastlineofcell#1:
#makethegraphsnicer
pd.set_option('display.mpl_style','default')
Itspurposeistomakethegraphswewilllookatinthissectionalittlebitprettier.
Okay,so,firstofallwehavetoinstructthenotebookthatwewanttousematplotlib
inline.ThismeansthatwhenweaskPandastoplotsomething,wewillhavetheresult
renderedinthecelloutputframe.Inordertodothis,wejustneedonesimpleinstruction:
#31
%matplotlibinline
Youcanalsoinstructthenotebooktodothiswhenyoustartitfromtheconsolebypassing
aparameter,butIwantedtoshowyouthiswaytoo,sinceitcanbeannoyingtohaveto
restartthenotebookjustbecauseyouwanttoplotsomething.Inthisway,youcandoiton
theflyandthenkeepworking.
Next,we’regoingtosetsomeparametersonpylab.Thisisforplottingpurposesandit
willremoveawarningthatafonthasn’tbeenfound.Isuggestthatyoudonotexecutethis
lineandkeepgoing.Ifyougetawarningthatafontismissing,comebacktothiscelland
runit.
#32
importpylab
pylab.rcParams.update({'font.family':'serif'})
ThisbasicallytellsPylabtousethefirstavailableseriffont.Itissimplebuteffective,and
youcanexperimentwithotherfontstoo.
NowthattheDataFrameiscomplete,let’srundf.describe()(#33)again.Theresults
shouldlooksomethinglikethis:
WOW! eBook
www.wowebook.org
Thiskindofquickresultisperfecttosatisfythosemanagerswhohave20secondsto
dedicatetoyouandjustwantroughnumbers.
Note
Onceagain,pleasekeepinmindthatourcampaignshavedifferentcurrencies,sothese
numbersareactuallymeaningless.ThepointhereistodemonstratetheDataFrame
capabilities,nottogettoacorrectordetailedanalysisofrealdata.
Alternatively,agraphisusuallymuchbetterthanatablewithnumbersbecauseit’smuch
easiertoreaditanditgivesyouimmediatefeedback.So,let’sgraphoutthefourpiecesof
informationwehaveoneachcampaign:budget,spent,clicks,andimpressions.
#34
df[['Budget','Spent','Clicks','Impressions']].hist(
bins=16,figsize=(16,6));
Weextrapolatethosefourcolumns(thiswillgiveusanotherDataFramemadewithonly
thosecolumns)andcallthehistogramhist()methodonit.Wegivesomemeasurements
onthebinsandfiguresizes,butbasicallyeverythingisdoneautomatically.
Oneimportantthing:sincethisinstructionistheonlyoneinthiscell(whichalsomeans,
it’sthelastone),thenotebookwillprintitsresultbeforedrawingthegraph.Tosuppress
thisbehaviorandhaveonlythegraphdrawnwithnoprinting,justaddasemicolonatthe
end(youthoughtIwasreminiscingaboutJava,didn’tyou?).Herearethegraphs:
WOW! eBook
www.wowebook.org
Theyarebeautiful,aren’tthey?Didyounoticetheseriffont?Howaboutthemeaningof
thosefigures?Ifyougobackto#6andtakealookatthewaywegeneratethedata,you
willseethatallthesegraphsmakeperfectsense.
Budgetissimplyarandomintegerinaninterval,thereforewewereexpectingauniform
distribution,andtherewehaveit;it’spracticallyaconstantline.
Spentisauniformdistributionaswell,butthehighendofitsintervalisthebudget,which
ismoving,thismeansweshouldexpectsomethinglikeaquadratichyperbolethat
decreasestotheright.Andthereitisaswell.
Clickswasgeneratedwithatriangulardistributionwithmeanroughly20%oftheinterval
size,andyoucanseethatthepeakisrightthere,atabout20%totheleft.
Finally,ImpressionswasaGaussiandistribution,whichistheonethatassumesthe
famousbellshape.Themeanwasexactlyinthemiddleandwehadstandarddeviationof
2.Youcanseethatthegraphmatchesthoseparameters.
Good!Let’splotoutthemeasureswecalculated:
#35
df[['CTR','CPC','CPI']].hist(
bins=20,figsize=(16,6));
WOW! eBook
www.wowebook.org
Wecanseethatthecostperclickishighlyskewedtotheleft,meaningthatmostofthe
CPCvaluesareverylow.Thecostperimpressionhasasimilarshape,butlessextreme.
Now,allthisisnice,butifyouwantedtoanalyzeonlyaparticularsegmentofthedata,
howwouldyoudoit?WecanapplyamasktoaDataFrame,sothatwegetanotherone
withonlytherowsthatsatisfythemaskcondition.It’slikeapplyingaglobalrow-wiseif
clause.
#36
mask=(df.Spent>0.75*df.Budget)
df[mask][['Budget','Spent','Clicks','Impressions']].hist(
bins=15,figsize=(16,6),color='g');
Inthiscase,Ipreparedamasktofilteroutalltherowsforwhichthespentislessthanor
equalto75%ofthebudget.Inotherwords,we’llincludeonlythosecampaignsforwhich
wehavespentatleastthreequartersofthebudget.NoticethatinthemaskIamshowing
youanalternativewayofaskingforaDataFramecolumn,byusingdirectpropertyaccess
(object.property_name),insteadofdict-likeaccess(object['property_name']).If
property_nameisavalidPythonname,youcanusebothwaysinterchangeably
(JavaScriptworkslikethisaswell).
Themaskisappliedinthesamewaythatweaccessadictwithakey.Whenyouapplya
masktoaDataFrame,yougetbackanotheroneandweselectonlytherelevantcolumns
onthis,andcallhist()again.Thistime,justforfun,wewanttheresultstobepainted
green:
WOW! eBook
www.wowebook.org
Notethattheshapesofthegraphshaven’tchangedmuch,apartfromthespent,whichis
quitedifferent.Thereasonforthisisthatwe’veaskedonlyfortherowswherespentisat
least75%ofthebudget.Thismeansthatwe’reincludingonlytherowswherespentis
closetothebudget.Thebudgetnumberscomefromauniformdistribution.Therefore,itis
quiteobviousthatthespentisnowassumingthatkindofshape.Ifyoumaketheboundary
eventighter,andaskfor85%ormore,you’llseespentbecomemoreandmorelike
budget.
Let’snowaskforsomethingdifferent.Howaboutthemeasureofspent,click,and
impressionsgroupedbydayoftheweek?
#37
df_weekday=df.groupby(['DayofWeek']).sum()
df_weekday[['Impressions','Spent','Clicks']].plot(
figsize=(16,6),subplots=True);
ThefirstlinecreatesanewDataFrame,df_weekday,byaskingforagroupingby'Dayof
Week'ondf.Thefunctionusedtoaggregatethedataisaddition.
Thesecondlinegetsasliceofdf_weekdayusingalistofcolumnnames,somethingwe’re
accustomedtobynow.Ontheresultwecallplot(),whichisabitdifferenttohist().
Theoptionsubplots=Truemakesplotdrawthreeindependentgraphs:
WOW! eBook
www.wowebook.org
Interestinglyenough,wecanseethatmostoftheactionhappensonThursdays.Ifthis
weremeaningfuldata,thiswouldpotentiallybeimportantinformationtogivetoour
clients,andthisisthereasonI’mshowingyouthisexample.
Notethatthedaysaresortedalphabetically,whichscramblesthemupabit.Canyouthink
ofaquicksolutionthatwouldfixtheissue?I’llleaveittoyouasanexercisetocomeup
withsomething.
Let’sfinishthispresentationsectionwithacouplemorethings.First,asimple
aggregation.Wewanttoaggregateon'TargetGender'and'TargetAge',andshow
'Impressions'and'Spent'.Forboth,wewanttoseethemeanandthestandard
deviation.
#38
agg_config={
'Impressions':{
'MeanImpr':'mean',
'StdImpr':'std',
},
'Spent':['mean','std'],
}
df.groupby(['TargetGender','TargetAge']).agg(agg_config)
It’sveryeasytodoit.Wewillprepareadictionarythatwe’lluseasaconfiguration.I’m
showingyoutwooptionstodoit.Weuseanicerformatfor'Impressions',wherewe
passanesteddictwithdescription/functionaskey/valuepairs.Ontheotherhand,for
'Spent',wejustuseasimplerlistwithjustthefunctionnames.
Then,weperformagroupingonthe'TargetGender'and'TargetAge'columns,and
wepassourconfigurationdicttotheagg()method.Theresultistruncatedandrearranged
alittlebittomakeitfit,andshownhere:
ImpressionsSpent
WOW! eBook
www.wowebook.org
MeanImprStdImprmeanstd
TargetTarget
GenderAge
B20-255000002.189102239882209442.168488
20-305000002.245317271285236854.155720
20-355000001.886396243725174268.898935
20-404999992.100786247740211540.133771
20-455000001.772811148712118603.932051
...............
M20-255000002.022023212520215857.323228
20-305000002.111882292577231663.713956
20-354999991.965177255651222790.960907
20-404999991.932473282515250023.393334
20-454999991.905746271077219901.462405
Thisisthetextualrepresentation,ofcourse,butyoucanalsohavetheHTMLone.You
canseethatSpenthasthemeanandstdcolumnswhoselabelsaresimplythefunction
names,whileImpressionsfeaturesthenicetitlesweaddedtotheconfigurationdict.
Let’sdoonemorethingbeforewewrapthischapterup.Iwanttoshowyousomething
calledapivottable.It’skindofabuzzwordinthedataenvironment,soanexamplesuch
asthisone,albeitverysimple,isamust.
#39
pivot=df.pivot_table(
values=['Impressions','Clicks','Spent'],
index=['TargetAge'],
columns=['TargetGender'],
aggfunc=np.sum
)
pivot
Wecreateapivottablethatshowsusthecorrelationbetweenthetargetageand
impressions,clicks,andspent.Theselastthreewillbesubdividedaccordingtothetarget
gender.Theaggregationfunctionusedtocalculatetheresultsisthenumpy.sumfunction
(numpy.meanwouldbethedefault,hadInotspecifiedanything).
Aftercreatingthepivottable,wesimplyprintitwiththelastlineinthecell,andhere’sa
cropoftheresult:
WOW! eBook
www.wowebook.org
It’sprettyclearandprovidesveryusefulinformationwhenthedataismeaningful.
That’sit!I’llleaveyoutodiscovermoreaboutthewonderfulworldofIPython,Jupyter,
anddatascience.Istronglyencourageyoutogetcomfortablewiththenotebook
environment.It’smuchbetterthanaconsole,it’sextremelypracticalandfuntouse,and
youcanevendoslidesanddocumentswithit.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Wheredowegofromhere?
Datascienceisindeedafascinatingsubject.AsIsaidintheintroduction,thosewhowant
todelveintoitsmeandersneedtobewelltrainedinmathematicsandstatistics.Working
withdatathathasbeeninterpolatedincorrectlyrendersanyresultaboutituseless.The
samegoesfordatathathasbeenextrapolatedincorrectlyorsampledwiththewrong
frequency.Togiveyouanexample,imagineapopulationofindividualsthatarealignedin
aqueue.If,forsomereason,thegenderofthatpopulationalternatedbetweenmaleand
female,thequeuewouldbesomethinglikethis:F-M-F-M-F-M-F-M-F…
Ifyousampledittakingonlytheevenelements,youwoulddrawtheconclusionthatthe
populationwasmadeuponlyofmales,whilesamplingtheoddoneswouldtellyou
exactlytheopposite.
Ofcourse,thiswasjustasillyexample,Iknow,butbelievemeit’sveryeasytomake
mistakesinthisfield,especiallywhendealingwithbigdatawheresamplingismandatory
andtherefore,thequalityoftheintrospectionyoumakedepends,firstandforemost,onthe
qualityofthesamplingitself.
WhenitcomestodatascienceandPython,thesearethemaintoolsyouwanttolookat:
NumPy(http://www.numpy.org/):Thisisthefundamentalpackageforscientific
computingwithPython.ItcontainsapowerfulN-dimensionalarrayobject,
sophisticated(broadcasting)functions,toolsforintegratingC/C++andFortrancode,
usefullinearalgebra,Fouriertransform,randomnumbercapabilities,andmuch
more.
Scikit-Learn(http://scikit-learn.org/stable/):Thisisprobablythemostpopular
machinelearninglibraryinPython.Ithassimpleandefficienttoolsfordatamining
anddataanalysis,accessibletoeverybody,andreusableinvariouscontexts.It’sbuilt
onNumPy,SciPy,andMatplotlib.
Pandas(http://pandas.pydata.org/):Thisisanopensource,BSD-licensedlibrary
providinghigh-performance,easy-to-usedatastructures,anddataanalysistools.
We’veuseditthroughoutthiswholechapter.
IPython(http://ipython.org/)/Jupyter(http://jupyter.org/):Theseprovidearich
architectureforinteractivecomputing.
Matplotlib(http://matplotlib.org/):ThisisaPython2Dplottinglibrarythatproduces
publication-qualityfiguresinavarietyofhardcopyformatsandinteractive
environmentsacrossplatforms.MatplotlibcanbeusedinPythonscripts,thePython
andIPythonshellandnotebook,webapplicationservers,andsixgraphicaluser
interfacetoolkits.
Numba(http://numba.pydata.org/):Thisgivesyouthepowertospeedupyour
applicationswithhighperformancefunctionswrittendirectlyinPython.Withafew
annotations,array-orientedandmath-heavyPythoncodecanbejust-in-time
compiledtonativemachineinstructions,similarinperformancetoC,C++,and
Fortran,withouthavingtoswitchlanguagesorPythoninterpreters.
Bokeh(http://bokeh.pydata.org/en/latest/):It’saPython-interactivevisualization
WOW! eBook
www.wowebook.org
librarythattargetsmodernwebbrowsersforpresentation.Itsgoalistoprovide
elegant,conciseconstructionofnovelgraphicsinthestyleofD3.js,butalsodeliver
thiscapabilitywithhigh-performanceinteractivityoververylargeorstreaming
datasets.
Otherthanthesesinglelibraries,youcanalsofindecosystemssuchasSciPy
(http://scipy.org/)andAnaconda(https://www.continuum.io/),whichbundleseveral
differentpackagesinordertogiveyousomethingthatjustworksinan“out-of-the-box”
fashion.
Installingallthesetoolsandtheirseveraldependenciesishardonsomesystems,soI
suggestthatyoutryoutecosystemsaswellandseeifyouarecomfortablewiththem.It
maybeworthit.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,wetalkedaboutdatascience.Ratherthanattemptingtoexplainanything
aboutthisextremelywidesubject,wedelvedintoaproject.Wefamiliarizedourselves
withtheJupyternotebook,andwithdifferentlibrariessuchasPandas,Matplotlib,NumPy.
Ofcourse,havingtocompressallthisinformationintoonesinglechaptermeansIcould
onlytouchbrieflyonthesubjectsIpresented.Ihopetheprojectwe’vegonethrough
togetherhasbeencomprehensiveenoughtogiveyouagoodideaaboutwhatcould
potentiallybetheworkflowyoumightfollowwhenworkinginthisfield.
Thenextchapterisdedicatedtowebdevelopment.So,makesureyouhaveabrowser
readyandlet’sgo!
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter10.WebDevelopmentDone
Right
“Don’tbelieveeverythingyoureadontheWeb.” —Confucius
Inthischapter,we’regoingtoworkonawebsitetogether.Byworkingonasmallproject,
myaimistoopenawindowforyoutotakeapeekonwhatwebdevelopmentis,along
withthemainconceptsandtoolsyoushouldknowifyouwanttobesuccessfulwithit.
WOW! eBook
www.wowebook.org
WhatistheWeb?
TheWorldWideWeb,orsimplyWeb,isawayofaccessinginformationthroughtheuse
ofamediumcalledtheInternet.TheInternetisahugenetworkofnetworks,a
networkinginfrastructure.Itspurposeistoconnectbillionsofdevicestogether,allaround
theglobe,sothattheycancommunicatewithoneanother.Informationtravelsthroughthe
Internetinarichvarietyoflanguagescalledprotocols,whichallowdifferentdevicesto
speakthesametongueinordertosharecontent.
TheWebisaninformation-sharingmodel,builtontopoftheInternet,whichemploysthe
HypertextTransferProtocol(HTTP)asabasisfordatacommunication.TheWeb,
therefore,isjustoneoftheseveraldifferentwaysinformationcanbeexchangedoverthe
Internet:e-mail,instantmessaging,newsgroups,andsoon,theyallrelyondifferent
protocols.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
HowdoestheWebwork?
Inanutshell,HTTPisanasymmetricrequest-responseclient-serverprotocol.AnHTTP
clientsendsarequestmessagetoanHTTPserver.Theserver,inturn,returnsaresponse
message.Inotherwords,HTTPisapullprotocolinwhichtheclientpullsinformation
fromtheserver(asopposedtoapushprotocolinwhichtheserverpushesinformation
downtotheclient).Takealookatthefollowingimage:
HTTPisbasedonTCP/IP(TransmissionControlProtocol/InternetProtocol),which
providesthetoolsforareliablecommunicationexchange.
AnimportantfeatureoftheHTTPprotocolisthatit’sstateless.Thismeansthatthe
currentrequesthasnoknowledgeaboutwhathappenedinpreviousrequests.Thisisa
limitation,butyoucanbrowseawebsitewiththeillusionofbeingloggedin.Underthe
coversthough,whathappensisthat,onlogin,atokenofuserinformationissaved(most
oftenontheclientside,inspecialfilescalledcookies)sothateachrequesttheusermakes
carriesthemeansfortheservertorecognizetheuserandprovideacustominterfaceby
showingtheirname,keepingtheirbasketpopulated,andsoon.
Eventhoughit’sveryinteresting,we’renotgoingtodelveintotherichdetailsofHTTP
andhowitworks.However,we’regoingtowriteasmallwebsite,whichmeanswe’llhave
towritethecodetohandleHTTPrequestsandreturnHTTPresponses.Iwon’tkeep
prependingHTTPtothetermsrequestandresponsefromnowon,asItrusttherewon’tbe
anyconfusion.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
TheDjangowebframework
Forourproject,we’regoingtouseoneofthemostpopularwebframeworksyoucanfind
inthePythonecosystem:Django.
Awebframeworkisasetoftools(libraries,functions,classes,andsoon)thatwecanuse
tocodeawebsite.Weneedtodecidewhatkindofrequestswewanttoallowtobeissued
againstourwebserverandhowwerespondtothem.Awebframeworkistheperfecttool
todothatbecauseittakescareofmanythingsforussothatwecanconcentrateonlyon
theimportantbitswithouthavingtoreinventthewheel.
Note
Therearedifferenttypesofframeworks.Notallofthemaredesignedforwritingcodefor
theweb.Ingeneral,aframeworkisatoolthatprovidesfunctionalitiestofacilitatethe
developmentofsoftwareapplications,productsandsolutions.
WOW! eBook
www.wowebook.org
Djangodesignphilosophy
Djangoisdesignedaccordingtothefollowingprinciples:
DRY:Asin,Don’tRepeatYourself.Don’trepeatcode,andcodeinawaythat
makestheframeworkdeduceasmuchaspossiblefromaslittleaspossible.
Loosecoupling:Thevariouslayersoftheframeworkshouldn’tknowabouteach
other(unlessabsolutelynecessaryforwhateverreason).Loosecouplingworksbest
whenparalleledwithhighcohesion.ToquoteRobertMartin:puttingtogetherthings
whichchangeforthesamereason,andspreadingapartthosewhichchangefor
differentreasons.
Lesscode:Applicationsshouldusetheleastpossibleamountofcode,andbewritten
inawaythatfavorsreuseasmuchaspossible.
Consistency:WhenusingtheDjangoframework,regardlessofwhichlayeryou’re
codingagainst,yourexperiencewillbeveryconsistentwiththedesignpatternsand
paradigmsthatwerechosentolayouttheproject.
Theframeworkitselfisdesignedaroundthemodel-template-view(MTV)pattern,which
isavariantofmodel-view-controller(MVC),whichiswidelyemployedbyother
frameworks.Thepurposeofsuchpatternsistoseparateconcernsandpromotecodereuse
andquality.
Themodellayer
Ofthethreelayers,thisistheonethatdefinesthestructureofthedatathatishandledby
theapplication,anddealswithdatasources.Amodelisaclassthatrepresentsadata
structure.ThroughsomeDjangomagic,modelsaremappedtodatabasetablessothatyou
canstoreyourdatainarelationaldatabase.
Note
Arelationaldatabasestoresdataintablesinwhicheachcolumnisapropertyofthedata
andeachrowrepresentsasingleitemorentryinthecollectionrepresentedbythattable.
Throughtheprimarykeyofeachtable,whichisthatpartofthedatathatallowsto
uniquelyidentifyeachitem,itispossibletoestablishrelationshipsbetweenitems
belongingtodifferenttables,thatis,toputthemintorelation.
Thebeautyofthissystemisthatyoudon’thavetowritedatabase-specificcodeinorderto
handleyourdata.Youwilljusthavetoconfigureyourmodelscorrectlyandsimplyuse
them.TheworkonthedatabaseisdoneforyoubytheDjangoobject-relationalmapping
(ORM),whichtakescareoftranslatingoperationsdoneonPythonobjectsintoalanguage
thatarelationaldatabasecanunderstand:SQL(StructuredQueryLanguage).
Onebenefitofthisapproachisthatyouwillbeabletochangedatabaseswithoutrewriting
yourcodesinceallthedatabasespecificcodeisproducedbyDjangoonthefly,according
towhichdatabaseit’sconnectedto.RelationaldatabasesspeakSQL,buteachofthemhas
itsownuniqueflavorofit;therefore,nothavingtohardcodeanySQLinourapplicationis
atremendousadvantage.
WOW! eBook
www.wowebook.org
Djangoallowsyoutomodifyyourmodelsatanytime.Whenyoudo,youcanruna
commandthatcreatesamigration,whichisthesetofinstructionsneededtoportthe
databaseinastatethatrepresentsthecurrentdefinitionofyourmodels.
Tosummarize,thislayerdealswithdefiningthedatastructuresyouneedtohandleinyour
websiteandgivesyouthemeanstosaveandloadthemfromandtothedatabaseby
simplyaccessingthemodels,whicharePythonobjects.
Theviewlayer
Thefunctionofaviewishandlingarequest,performingwhateveractionneedstobe
carriedout,andeventuallyreturningaresponse.Forexample,ifyouopenyourbrowser
andrequestapagecorrespondingtoacategoryofproductsinane-commerceshop,the
viewwilllikelytalktothedatabase,askingforallthecategoriesthatarechildrenofthe
selectedcategory(forexample,todisplaytheminanavigationsidebar)andforallthe
productsthatbelongtotheselectedcategory,inordertodisplaythemonthepage.
Therefore,theviewisthemechanismthroughwhichwecanfulfillarequest.Itsresult,the
responseobject,canassumeseveraldifferentforms:aJSONpayload,text,anHTML
page,andsoon.Whenyoucodeawebsite,yourresponsesusuallyconsistofHTMLor
JSON.
Note
TheHypertextMarkupLanguage,orHTML,isthestandardmarkuplanguageusedto
createwebpages.WebbrowsersrunenginesthatarecapableofinterpretingHTMLcode
andrenderitintowhatweseewhenweopenapageofawebsite.
Thetemplatelayer
Thisisthelayerthatprovidesthebridgebetweenbackendandfrontenddevelopment.
WhenaviewhastoreturnHTML,itusuallydoesitbypreparingacontextobject(adict)
withsomedata,andthenitfeedsthiscontexttoatemplate,whichisrendered(thatisto
say,transformedintoHTML)andreturnedtothecallerintheformofaresponse(more
precisely,thebodyoftheresponse).Thismechanismallowsformaximumcodereuse.If
yougobacktothecategoryexample,it’seasytoseethat,ifyoubrowseawebsitethat
sellsproducts,itdoesn’treallymatterwhichcategoryyouclickonorwhattypeofsearch
youperform,thelayoutoftheproductspagedoesn’tchange.Whatdoeschangeisthedata
withwhichthatpageispopulated.
Therefore,thelayoutofthepageisdefinedbyatemplate,whichiswritteninamixtureof
HTMLandDjangotemplatelanguage.Theviewthatservesthatpagecollectsallthe
productstobedisplayedinthecontextdict,andfeedsittothetemplatewhichwillbe
renderedintoanHTMLpagebytheDjangotemplateengine.
WOW! eBook
www.wowebook.org
TheDjangoURLdispatcher
ThewayDjangoassociatesaUniformResourceLocator(URL)withaviewisthrough
matchingtherequestedURLwiththepatternsthatareregisteredinaspecialfile.AURL
representsapageinawebsiteso,forexample,http://mysite.com/categories?id=123
wouldprobablypointtothepageforthecategorywithID123onmywebsite,while
https://mysite.com/loginwouldprobablybetheuserloginpage.
Tip
ThedifferencebetweenHTTPandHTTPSisthatthelatteraddsencryptiontotheprotocol
sothatthedatathatyouexchangewiththewebsiteissecured.Whenyouputyourcredit
carddetailsonawebsite,orloginanywhere,ordoanythingaroundsensitivedata,you
wanttomakesurethatyou’reusingHTTPS.
Regularexpressions
ThewayDjangomatchesURLstopatternsisthrougharegularexpression.Aregular
expressionisasequenceofcharactersthatdefinesasearchpatternwithwhichwecan
carryoutoperationssuchaspatternandstringmatching,find/replace,andsoon.
Regularexpressionshaveaspecialsyntaxtoindicatethingslikedigits,letters,spaces,and
soon,aswellashowmanytimesweexpectacharactertoappear,andmuchmore.A
completeexplanationofthistopicisbeyondofthescopeofthebook.However,itisa
veryimportanttopic,sotheprojectwe’regoingtoworkontogetherwillevolvearoundit,
inthehopethatyouwillbestimulatedtofindthetimetoexploreitabitmoreonyour
own.
Togiveyouaquickexample,imaginethatyouwantedtospecifyapatterntomatchadate
suchas"26-12-1947".Thisstringconsistsoftwodigits,onedash,twodigits,onedash,
andfinallyfourdigits.Therefore,wecouldwriteitlikethis:r'[0-9]{2}-[0-9]{2}-[0-9]
{4}'.Wecreatedaclassbyusingsquarebrackets,andwedefinedarangeofdigitsinside,
from0to9,henceallthepossibledigits.Then,betweencurlybraces,wesaythatwe
expecttwoofthem.Thenadash,thenwerepeatthispatternonceasitis,andoncemore,
bychanginghowmanydigitsweexpect,andwithoutthefinaldash.Havingaclasslike
[0-9]issuchacommonpatternthataspecialnotationhasbeencreatedasashortcut:
'\d'.Therefore,wecanrewritethepatternlikethis:r'\d{2}-\d{2}-\d{4}'anditwill
workexactlythesame.Thatrinfrontofthestringstandsforraw,anditspurposeisto
alterthewayeverybackslash'\'isinterpretedbytheregularexpressionengine.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Aregexwebsite
So,hereweare.We’llcodeawebsitethatstoresregularexpressionssothatwe’llbeable
toplaywiththemalittlebit.
Note
Beforeweproceedcreatingtheproject,I’dliketospendawordaboutCSS.CSS
(CascadingStyleSheets)arefilesinwhichwespecifyhowthevariouselementsonan
HTMLpagelook.Youcansetallsortsofpropertiessuchasshape,size,color,margins,
borders,fonts,andsoon.Inthisproject,Ihavetriedmybesttoachieveadecentresulton
thepages,butI’mneitherafrontenddevelopernoradesigner,sopleasedon’tpaytoo
muchattentiontohowthingslook.Tryandfocusonhowtheywork.
WOW! eBook
www.wowebook.org
SettingupDjango
OntheDjangowebsite(https://www.djangoproject.com/),youcanfollowthetutorial,
whichgivesyouaprettygoodideaofDjango’scapabilities.Ifyouwant,youcanfollow
thattutorialfirstandthencomebacktothisexample.So,firstthingsfirst;let’sinstall
Djangoinyourvirtualenvironment:
$pipinstalldjango
Whenthiscommandisdone,youcantestitwithinaconsole(trydoingitwithbpython,it
givesyouashellsimilartoIPythonbutwithniceintrospectioncapabilities):
>>>importdjango
>>>django.VERSION
(1,8,4,'final',0)
NowthatDjangoisinstalled,we’regoodtogo.We’llhavetodosomescaffolding,soI’ll
quicklyguideyouthroughthat.
Startingtheproject
Chooseafolderinthebook’senvironmentandchangeintothat.I’llusech10.Fromthere,
westartaDjangoprojectwiththefollowingcommand:
$django-adminstartprojectregex
ThiswillpreparetheskeletonforaDjangoprojectcalledregex.Changeintotheregex
folderandrunthefollowing:
$pythonmanage.pyrunserver
Youshouldbeabletogotohttp://127.0.0.1:8000/withyourbrowserandseetheIt
worked!defaultDjangopage.Thismeansthattheprojectiscorrectlysetup.Whenyou’ve
seenthepage,killtheserverwithCtrl+C(orwhateveritsaysintheconsole).I’llpaste
thefinalstructurefortheprojectnowsothatyoucanuseitasareference:
$tree-Aregex#fromthech10folder
regex
├──db.sqlite3
├──entries
│├──admin.py
│├──forms.py
│├──__init__.py
│├──migrations
││├──0001_initial.py
││└──__init__.py
│├──models.py
│├──static
││└──entries
││└──css
││└──main.css
│├──templates
││└──entries
││├──base.html
WOW! eBook
www.wowebook.org
││├──footer.html
││├──home.html
││├──insert.html
││└──list.html
│└──views.py
├──manage.py
└──regex
├──__init__.py
├──settings.py
├──urls.py
└──wsgi.py
Don’tworryifyou’remissingfiles,we’llgetthere.ADjangoprojectistypicallya
collectionofseveraldifferentapplications.Eachapplicationismeanttoprovidea
functionalityinaself-contained,reusablefashion.We’llcreatejustone,calledentries:
$pythonmanage.pystartappentries
Withintheentriesfolderthathasbeencreated,youcangetridofthetests.pymodule.
Now,let’sfixtheregex/settings.pyfileintheregexfolder.Weneedtoaddour
applicationtotheINSTALLED_APPStuplesothatwecanuseit(additatthebottomofthe
tuple):
INSTALLED_APPS=(
...djangoapps…
'entries',
)
Then,youmaywanttofixthelanguageandtimezoneaccordingtoyourpersonal
preference.IliveinLondon,soIsetthemlikethis:
LANGUAGE_CODE='en-gb'
TIME_ZONE='Europe/London'
Thereisnothingelsetodointhisfile,soyoucansaveandcloseit.
Nowit’stimetoapplythemigrationstothedatabase.Djangoneedsdatabasesupportto
handleusers,sessions,andthingslikethat,soweneedtocreateadatabaseandpopulateit
withthenecessarydata.Luckily,thisisveryeasilydonewiththefollowingcommand:
$pythonmanage.pymigrate
Note
Forthisproject,weuseaSQLitedatabase,whichisbasicallyjustafile.Onarealproject,
youwouldprobablyuseadifferentdatabaseenginelikeMySQLorPostgreSQL.
Creatingusers
Nowthatwehaveadatabase,wecancreateasuperuserusingtheconsole.
$pythonmanage.pycreatesuperuser
Afterenteringusernameandotherdetails,wehaveauserwithadminprivileges.Thisis
enoughtoaccesstheDjangoadminsection,sotryandstarttheserver:
WOW! eBook
www.wowebook.org
$pythonmanage.pyrunserver
ThiswillstarttheDjangodevelopmentserver,whichisaveryusefulbuilt-inwebserver
thatyoucanusewhileworkingwithDjango.Nowthattheserverisrunning,wecan
accesstheadminpageathttp://localhost:8000/admin/.Iwillshowyouascreenshot
ofthissectionlater.Ifyouloginwiththecredentialsoftheuseryoujustcreatedandhead
totheAuthenticationandAuthorizationsection,you’llfindUsers.Openthatandyou
willbeabletoseethelistofusers.Youcaneditthedetailsofanyuseryouwantasan
admin.Inourcase,makesureyoucreateadifferentonesothatthereareatleasttwousers
inthesystem(we’llneedthemlater).I’llcallthefirstuserFabrizio(username:fab)and
thesecondoneAdriano(username:adri)inhonorofmyfather.
Bytheway,youshouldseethattheDjangoadminpanelcomesforfreeautomatically.You
defineyourmodels,hookthemup,andthat’sit.Thisisanincredibletoolthatshowshow
advancedDjango’sintrospectioncapabilitiesare.Moreover,itiscompletelycustomizable
andextendable.It’strulyanexcellentpieceofwork.
WOW! eBook
www.wowebook.org
AddingtheEntrymodel
Nowthattheboilerplateisoutoftheway,andwehaveacoupleofusers,we’rereadyto
code.WestartbyaddingtheEntrymodeltoourapplicationsothatwecanstoreobjectsin
thedatabase.Here’sthecodeyou’llneedtoadd(remembertousetheprojecttreefor
reference):
entries/models.py
fromdjango.dbimportmodels
fromdjango.contrib.auth.modelsimportUser
fromdjango.utilsimporttimezone
classEntry(models.Model):
user=models.ForeignKey(User)
pattern=models.CharField(max_length=255)
test_string=models.CharField(max_length=255)
date_added=models.DateTimeField(default=timezone.now)
classMeta:
verbose_name_plural='entries'
Thisisthemodelwe’llusetostoreregularexpressionsinoursystem.We’llstorea
pattern,ateststring,areferencetotheuserwhocreatedtheentry,andthemomentof
creation.Youcanseethatcreatingamodelisactuallyquiteeasy,butnonetheless,let’sgo
throughitlinebyline.
Firstweneedtoimportthemodelsmodulefromdjango.db.Thiswillgiveusthebase
classforourEntrymodel.Djangomodelsarespecialclassesandmuchisdoneforus
behindthesceneswhenweinheritfrommodels.Model.
Wewantareferencetotheuserwhocreatedtheentry,soweneedtoimporttheUser
modelfromDjango’sauthorizationapplicationandwealsoneedtoimportthetimezone
modeltogetaccesstothetimezone.now()function,whichprovidesuswithatimezoneawareversionofdatetime.now().Thebeautyofthisisthatit’shookedupwiththe
TIME_ZONEsettingsIshowedyoubefore.
Asfortheprimarykeyforthisclass,ifwedon’tsetoneexplicitly,Djangowilladdonefor
us.AprimarykeyisakeythatallowsustouniquelyidentifyanEntryobjectinthe
database(inthiscase,Djangowilladdanauto-incrementingintegerID).
So,wedefineourclass,andwesetupfourclassattributes.WehaveaForeignKey
attributethatisourreferencetotheUsermodel.WealsohavetwoCharFieldattributes
thatholdthepatternandteststringsforourregularexpressions.Wealsohavea
DateTimeField,whosedefaultvalueissettotimezone.now.Notethatwedon’tcall
timezone.nowrightthere,it’snow,notnow().So,we’renotpassingaDateTimeinstance
(setatthemomentintimewhenthatlineisparsed)rather,we’repassingacallable,a
functionthatiscalledwhenwesaveanentryinthedatabase.Thisissimilartothe
callbackmechanismweusedinChapter8,TheEdges–GUIsandScripts,whenwewere
assigningcommandstobuttonclicks.
WOW! eBook
www.wowebook.org
Thelasttwolinesareveryinteresting.WedefineaclassMetawithintheEntryclassitself.
TheMetaclassisusedbyDjangotoprovideallsortsofextrainformationforamodel.
Djangohasagreatdealoflogicunderthehoodtoadaptitsbehavioraccordingtothe
informationweputintheMetaclass.Inthiscase,intheadminpanel,thepluralized
versionofEntrywouldbeEntrys,whichiswrong,thereforeweneedtomanuallysetit.
Wespecifythepluralalllowercase,asDjangotakescareofcapitalizingitforuswhen
needed.
Nowthatwehaveanewmodel,weneedtoupdatethedatabasetoreflectthenewstateof
thecode.Inordertodothis,weneedtoinstructDjangothatitneedstocreatethecodeto
updatethedatabase.Thiscodeiscalledmigration.Let’screateitandexecuteit:
$pythonmanage.pymakemigrationsentries
$pythonmanage.pymigrate
Afterthesetwoinstructions,thedatabasewillbereadytostoreEntryobjects.
Note
Therearetwodifferentkindsofmigrations:dataandschemamigration.Datamigrations
portdatafromonestatetoanotherwithoutalteringitsstructure.Forexample,adata
migrationcouldsetallproductsforacategoryasoutofstockbyswitchingaflagtoFalse
or0.Aschemamigrationisasetofinstructionsthatalterthestructureofthedatabase
schema.Forexample,thatcouldbeaddinganagecolumntoaPersontable,orincreasing
themaximumlengthofafieldtoaccountforverylongaddresses.Whendevelopingwith
Django,it’squitecommontohavetoperformbothkindsofmigrationsoverthecourseof
development.Dataevolvescontinuously,especiallyifyoucodeinanagileenvironment.
WOW! eBook
www.wowebook.org
Customizingtheadminpanel
ThenextstepistohooktheEntrymodelupwiththeadminpanel.Youcandoitwithone
lineofcode,butinthiscase,Iwanttoaddsomeoptionstocustomizeabitthewaythe
adminpanelshowstheentries,bothinthelistviewofallentryitemsinthedatabaseand
intheformviewthatallowsustocreateandmodifythem.
Allweneedtodoistoaddthefollowingcode:
entries/admin.py
fromdjango.contribimportadmin
from.modelsimportEntry
@admin.register(Entry)
classEntryAdmin(admin.ModelAdmin):
fieldsets=[
('RegularExpression',
{'fields':['pattern','test_string']}),
('OtherInformation',
{'fields':['user','date_added']}),
]
list_display=('pattern','test_string','user')
list_filter=['user']
search_fields=['test_string']
Thisissimplybeautiful.Myguessisthatyouprobablyalreadyunderstandmostofit,even
ifyou’renewtoDjango.
So,westartbyimportingtheadminmoduleandtheEntrymodel.Becausewewantto
fostercodereuse,weimporttheEntrymodelusingarelativeimport(there’sadotbefore
models).Thiswillallowustomoveorrenametheappwithouttoomuchtrouble.Then,
wedefinetheEntryAdminclass,whichinheritsfromadmin.ModelAdmin.Thedecoration
ontheclasstellsDjangotodisplaytheEntrymodelintheadminpanel,andwhatweput
intheEntryAdminclasstellsDjangohowtocustomizethewayithandlesthismodel.
Firstly,wespecifythefieldsetsforthecreate/editpage.Thiswilldividethepageinto
twosectionssothatwegetabettervisualizationofthecontent(patternandteststring)
andtheotherdetails(userandtimestamp)separately.
Then,wecustomizethewaythelistpagedisplaystheresults.Wewanttoseeallthefields,
butnotthedate.Wealsowanttobeabletofilterontheusersothatwecanhavealistof
alltheentriesbyjustoneuser,andwewanttobeabletosearchontest_string.
Iwillgoaheadandaddthreeentries,oneformyselfandtwoonbehalfofmyfather.The
resultisshowninthenexttwoimages.Afterinsertingthem,thelistpagelookslikethis:
WOW! eBook
www.wowebook.org
IhavehighlightedthethreepartsofthisviewthatwecustomizedintheEntryAdminclass.
Wecanfilterbyuser,wecansearchandwehaveallthefieldsdisplayed.Ifyouclickona
pattern,theeditviewopensup.
Afterourcustomization,itlookslikethis:
Noticehowwehavetwosections:RegularExpressionandOtherInformation,thanks
toourcustomEntryAdminclass.Haveagowithit,addsomeentriestoacoupleof
differentusers,getfamiliarwiththeinterface.Isn’titnicetohaveallthisforfree?
WOW! eBook
www.wowebook.org
Creatingtheform
Everytimeyoufillinyourdetailsonawebpage,you’reinsertingdatainformfields.A
formisapartoftheHTMLDocumentObjectModel(DOM)tree.InHTML,you
createaformbyusingtheformtag.Whenyouclickonthesubmitbutton,yourbrowser
normallypackstheformdatatogetherandputsitinthebodyofaPOSTrequest.As
opposedtoGETrequests,whichareusedtoaskthewebserverforaresource,aPOST
requestnormallysendsdatatothewebserverwiththeaimofcreatingorupdatinga
resource.Forthisreason,handlingPOSTrequestsusuallyrequiresmorecarethanGET
requests.
WhentheserverreceivesdatafromaPOSTrequest,thatdataneedstobevalidated.
Moreover,theserverneedstoemploysecuritymechanismstoprotectagainstvarious
typesofattacks.Oneattackthatisverydangerousisthecross-siterequestforgery
(CSRF)attack,whichhappenswhendataissentfromadomainthatisnottheonethe
userisauthenticatedon.Djangoallowsyoutohandlethisissueinaveryelegantway.
So,insteadofbeinglazyandusingtheDjangoadmintocreatetheentries,I’mgoingto
showyouhowtodoitusingaDjangoform.Byusingthetoolstheframeworkgivesyou,
yougetaverygooddegreeofvalidationworkalreadydone(infact,wewon’tneedtoadd
anycustomvalidationourselves).
TherearetwokindsofformclassesinDjango:FormandModelForm.Youusetheformer
tocreateaformwhoseshapeandbehaviordependsonhowyoucodetheclass,whatfields
youadd,andsoon.Ontheotherhand,thelatterisatypeofformthat,albeitstill
customizable,infersfieldsandbehaviorfromamodel.SinceweneedaformfortheEntry
model,we’llusethatone.
entries/forms.py
fromdjango.formsimportModelForm
from.modelsimportEntry
classEntryForm(ModelForm):
classMeta:
model=Entry
fields=['pattern','test_string']
Amazinglyenough,thisisallwehavetodotohaveaformthatwecanputonapage.The
onlynotablethinghereisthatwerestrictthefieldstoonlypatternandtest_string.
Onlylogged-inuserswillbeallowedaccesstotheinsertpage,andthereforewedon’t
needtoaskwhotheuseris:weknowthat.Asforthedate,whenwesaveanEntry,the
date_addedfieldwillbesetaccordingtoitsdefault,thereforewedon’tneedtospecify
thataswell.We’llseeintheviewhowtofeedtheuserinformationtotheformbefore
saving.So,allthebackgroundworkisdone,allweneedistheviewsandthetemplates.
Let’sstartwiththeviews.
WOW! eBook
www.wowebook.org
Writingtheviews
Weneedtowritethreeviews.Weneedoneforthehomepage,onetodisplaythelistofall
entriesforauser,andonetocreateanewentry.Wealsoneedviewstologinandlogout.
ButthankstoDjango,wedon’tneedtowritethem.I’llpasteallthecode,andthenwe’ll
gothroughittogether,stepbystep.
entries/views.py
importre
fromdjango.contrib.auth.decoratorsimportlogin_required
fromdjango.contrib.messages.viewsimportSuccessMessageMixin
fromdjango.core.urlresolversimportreverse_lazy
fromdjango.utils.decoratorsimportmethod_decorator
fromdjango.views.genericimportFormView,TemplateView
from.formsimportEntryForm
from.modelsimportEntry
classHomeView(TemplateView):
template_name='entries/home.html'
@method_decorator(
login_required(login_url=reverse_lazy('login')))
defget(self,request,*args,**kwargs):
context=self.get_context_data(**kwargs)
returnself.render_to_response(context)
classEntryListView(TemplateView):
template_name='entries/list.html'
@method_decorator(
login_required(login_url=reverse_lazy('login')))
defget(self,request,*args,**kwargs):
context=self.get_context_data(**kwargs)
entries=Entry.objects.filter(
user=request.user).order_by('-date_added')
matches=(self._parse_entry(entry)forentryinentries)
context['entries']=list(zip(entries,matches))
returnself.render_to_response(context)
def_parse_entry(self,entry):
match=re.search(entry.pattern,entry.test_string)
ifmatchisnotNone:
return(
match.group(),
match.groups()orNone,
match.groupdict()orNone
)
returnNone
classEntryFormView(SuccessMessageMixin,FormView):
template_name='entries/insert.html'
form_class=EntryForm
success_url=reverse_lazy('insert')
success_message="Entrywascreatedsuccessfully"
WOW! eBook
www.wowebook.org
@method_decorator(
login_required(login_url=reverse_lazy('login')))
defget(self,request,*args,**kwargs):
returnsuper(EntryFormView,self).get(
request,*args,**kwargs)
@method_decorator(
login_required(login_url=reverse_lazy('login')))
defpost(self,request,*args,**kwargs):
returnsuper(EntryFormView,self).post(
request,*args,**kwargs)
defform_valid(self,form):
self._save_with_user(form)
returnsuper(EntryFormView,self).form_valid(form)
def_save_with_user(self,form):
self.object=form.save(commit=False)
self.object.user=self.request.user
self.object.save()
Let’sstartwiththeimports.Weneedtheremoduletohandleregularexpressions,thenwe
needafewclassesandfunctionsfromDjango,andfinally,weneedtheEntrymodeland
theEntryFormform.
Thehomeview
ThefirstviewisHomeView.ItinheritsfromTemplateView,whichmeansthattheresponse
willbecreatedbyrenderingatemplatewiththecontextwe’llcreateintheview.Allwe
havetodoisspecifythetemplate_nameclassattributetopointtothecorrecttemplate.
Djangopromotescodereusetoapointthatifwedidn’tneedtomakethisviewaccessible
onlytologged-inusers,thefirsttwolineswouldhavebeenallweneeded.
However,wewantthisviewtobeaccessibleonlytologged-inusers;therefore,weneedto
decorateitwithlogin_required.Now,historicallyviewsinDjangousedtobefunctions;
therefore,thisdecoratorwasdesignedtoacceptafunctionnotamethodlikewehavein
thisclass.We’reusingDjangoclass-basedviewsinthisprojectso,inordertomakethings
work,weneedtotransformlogin_requiredsothatitacceptsamethod(thedifference
beinginthefirstargument:self).Wedothisbypassinglogin_requiredto
method_decorator.
Wealsoneedtofeedthelogin_requireddecoratorwithlogin_urlinformation,andhere
comesanotherwonderfulfeatureofDjango.Asyou’llseeafterwe’redonewiththe
views,inDjango,youtieaviewtoaURLthroughapattern,consistingofaregular
expressionandotherinformation.Youcangiveanametoeachentryintheurls.pyfileso
thatwhenyouwanttorefertoaURL,youdon’thavetohardcodeitsvalueintoyourcode.
AllyouhavetodoisgetDjangotoreverse-engineerthatURLfromthenamewegaveto
theentryinurls.pydefiningtheURLandtheviewthatistiedtoit.Thismechanismwill
becomeclearerlater.Fornow,justthinkofreverse('...')asawayofgettingaURL
fromanidentifier.Inthisway,youonlywritetheactualURLonce,intheurls.pyfile,
WOW! eBook
www.wowebook.org
whichisbrilliant.Intheviews.pycode,weneedtousereverse_lazy,whichworks
exactlylikereversewithonemajordifference:itonlyfindstheURLwhenweactually
needit(inalazyfashion).Thisisneededwhentheurls.pyfilehasn’tbeenloadedyet
whenthereversefunctionisused.
Thegetmethod,whichwejustdecorated,simplycallsthegetmethodoftheparentclass.
Ofcourse,thegetmethodisthemethodthatDjangocallswhenaGETrequestis
performedagainsttheURLtiedtothisview.
Theentrylistview
Thisviewismuchmoreinterestingthanthepreviousone.Firstofall,wedecoratetheget
methodaswedidbefore.Insideofit,weneedtopreparealistofEntryobjectsandfeedit
tothetemplate,whichshowsittotheuser.Inordertodoso,westartbygettingthe
contextdictlikewe’resupposedtodo,bycallingtheget_context_datamethodofthe
TemplateViewclass.Then,weusetheORMtogetalistoftheentries.Wedothisby
accessingtheobjectsmanager,andcallingafilteronit.Wefiltertheentriesaccordingto
whichuserisloggedin,andweaskforthemtobesortedinadescendingorder(that'-'
infrontofthenamespecifiesthedescendingorder).Theobjectsmanageristhedefault
managereveryDjangomodelisaugmentedwithoncreation,itallowsustointeractwith
thedatabasethroughitsmethods.
Weparseeachentrytogetalistofmatches(actually,Icodeditsothatmatchesisa
generatorexpression).Finally,weaddtothecontextan'entries'keywhosevalueisthe
couplingofentriesandmatches,sothateachEntryinstanceispairedwiththeresulting
matchofitspatternandteststring.
Onthelastline,wesimplyaskDjangotorenderthetemplateusingthecontextwe
created.
Takealookatthe_parse_entrymethod.Allitdoesisperformasearchonthe
entry.test_stringwiththeentry.pattern.IftheresultingmatchobjectisnotNone,it
meansthatwefoundsomething.Ifso,wereturnatuplewiththreeelements:theoverall
group,thesubgroups,andthegroupdictionary.Ifyou’renotfamiliarwiththeseterms,
don’tworry,you’llseeascreenshotsoonwithanexample.WereturnNoneifthereisno
match.
Theformview
Finally,let’sexamineEntryFormView.Thisisparticularlyinterestingforafewreasons.
Firstly,itshowsusaniceexampleofPython’smultipleinheritance.Wewanttodisplaya
messageonthepage,afterhavinginsertedanEntry,soweinheritfrom
SuccessMessageMixin.Butwewanttohandleaformaswell,sowealsoinheritfrom
FormView.
Note
Notethat,whenyoudealwithmixinsandinheritance,youmayhavetoconsidertheorder
inwhichyouspecifythebaseclassesintheclassdeclaration.
WOW! eBook
www.wowebook.org
Inordertosetupthisviewcorrectly,weneedtospecifyafewattributesatthebeginning:
thetemplatetoberendered,theformclasstobeusedtohandlethedatafromthePOST
request,theURLweneedtoredirecttheusertointhecaseofsuccess,andthesuccess
message.
AnotherinterestingfeatureisthatthisviewneedstohandlebothGETandPOSTrequests.
Whenwelandontheformpageforthefirsttime,theformisempty,andthatistheGET
request.Ontheotherhand,whenwefillintheformandwanttosubmittheEntry,we
makeaPOSTrequest.Youcanseethatthebodyofgetisconceptuallyidenticalto
HomeView.Djangodoeseverythingforus.
Thepostmethodisjustlikeget.Theonlyreasonweneedtocodethesetwomethodsis
sothatwecandecoratethemtorequirelogin.
WithintheDjangoformhandlingprocess(intheFormViewclass),thereareafewmethods
thatwecanoverrideinordertocustomizetheoverallbehavior.Weneedtodoitwiththe
form_validmethod.Thismethodwillbecalledwhentheformvalidationissuccessful.Its
purposeistosavetheformsothatanEntryobjectiscreatedoutofit,andthenstoredin
thedatabase.
Theonlyproblemisthatourformismissingtheuser.Weneedtointerceptthatmomentin
thechainofcallsandputtheuserinformationinourselves.Thisisdonebycallingthe
_save_with_usermethod,whichisverysimple.
Firstly,weaskDjangotosavetheformwiththecommitargumentsettoFalse.This
createsanEntryinstancewithoutattemptingtosaveittothedatabase.Savingit
immediatelywouldfailbecausetheuserinformationisnotthere.
ThenextlineupdatestheEntryinstance(self.object),addingtheuserinformationand,
onthelastline,wecansafelysaveit.ThereasonIcalleditobjectandsetitonthe
instancelikethatwastofollowwhattheoriginalFormViewclassdoes.
We’refiddlingwiththeDjangomechanismhere,soifwewantthewholethingtowork,
weneedtopayattentiontowhenandhowwemodifyitsbehavior,andmakesurewe
don’talteritincorrectly.Forthisreason,it’sveryimportanttoremembertocallthe
form_validmethodofthebaseclass(weusesuperforthat)attheendofourown
customizedversion,tomakesurethateveryotheractionthatmethodusuallyperformsis
carriedoutcorrectly.
Notehowtherequestistiedtoeachviewinstance(self.request)sothatwedon’tneed
topassitthroughwhenwerefactorourlogicintomethods.Notealsothattheuser
informationhasbeenaddedtotherequestautomaticallybyDjango.Finally,notethatthe
reasonwhyalltheprocessissplitintoverysmallmethodsliketheseissothatwecanonly
overridethosethatweneedtocustomize.Allthisremovestheneedtowritealotofcode.
Nowthatwehavetheviewscovered,let’sseehowwecouplethemtotheURLs.
WOW! eBook
www.wowebook.org
TyingupURLsandviews
Intheurls.pymodule,wetieeachviewtoaURL.Therearemanywaysofdoingthis.I
chosethesimplestone,whichworksperfectlyfortheextentofthisexercise,butyoumay
wanttoexplorethisargumentmoredeeplyifyouintendtoworkwithDjango.Thisisthe
corearoundwhichthewholewebsitelogicwillrevolve;therefore,youshouldtrytogetit
downcorrectly.Notethattheurls.pymodulebelongstotheprojectfolder.
regex/urls.py
fromdjango.conf.urlsimportinclude,url
fromdjango.contribimportadmin
fromdjango.contrib.authimportviewsasauth_views
fromdjango.core.urlresolversimportreverse_lazy
fromentries.viewsimportHomeView,EntryListView,EntryFormView
urlpatterns=[
url(r'^admin/',include(admin.site.urls)),
url(r'^entries/$',EntryListView.as_view(),name='entries'),
url(r'^entries/insert$',
EntryFormView.as_view(),
name='insert'),
url(r'^login/$',
auth_views.login,
kwargs={'template_name':'admin/login.html'},
name='login'),
url(r'^logout/$',
auth_views.logout,
kwargs={'next_page':reverse_lazy('home')},
name='logout'),
url(r'^$',HomeView.as_view(),name='home'),
]
Asyoucansee,themagiccomesfromtheurlfunction.Firstly,wepassitaregular
expression;thentheview;andfinally,aname,whichiswhatwewilluseinthereverse
andreverse_lazyfunctionstorecovertheURL.
Notethat,whenusingclass-basedviews,wehavetotransformthemintofunctions,which
iswhaturlisexpecting.Todothat,wecalltheas_view()methodonthem.
Notealsothatthefirsturlentry,fortheadmin,isspecial.InsteadofspecifyingaURL
andaview,itspecifiesaURLprefixandanotherurls.pymodule(fromtheadmin.site
package).Inthisway,DjangowillcompletealltheURLsfortheadminsectionby
prepending'admin/'toalltheURLsspecifiedinadmin.site.urls.Wecouldhavedone
thesameforourentriesapp(andweshouldhave),butIfeelitwouldhavebeenabittoo
muchforthissimpleproject.
Intheregularexpressionlanguage,the'^'and'$'symbolsrepresentthestartandendof
astring.Notethatifyouusetheinclusiontechnique,asfortheadmin,the'$'ismissing.
Ofcourse,thisisbecause'admin/'isjustaprefix,whichneedstobecompletedbyallthe
definitionsintheincludedurlsmodule.
WOW! eBook
www.wowebook.org
Somethingelseworthnoticingisthatwecanalsoincludethestringifiedversionofapath
toaview,whichwedofortheloginandlogoutviews.Wealsoaddinformationabout
whichtemplatestousewiththekwargsargument.Theseviewscomestraightfromthe
django.contrib.authpackage,bytheway,sothatwedon’tneedtowriteasinglelineof
codetohandleauthentication.Thisisbrilliantandsavesusalotoftime.
Eachurldeclarationmustbedonewithintheurlpatternslistandonthismatter,it’s
importanttoconsiderthat,whenDjangoistryingtofindaviewforaURLthathasbeen
requested,thepatternsareexercisedinorder,fromtoptobottom.Thefirstonethat
matchesistheonethatwillprovidetheviewforitso,ingeneral,youhavetoputspecific
patternsbeforegenericones,otherwisetheywillnevergetachancetobecaught.For
example,'^shop/categories/$'needstocomebefore'^shop'(notetheabsenceofthe
'$'inthelatter),otherwiseitwouldneverbecalled.Ourexamplefortheentriesworks
finebecauseIthoroughlyspecifiedURLsusingthe'$'attheend.
So,models,forms,admin,viewsandURLsarealldone.Allthatislefttodoistakecare
ofthetemplates.I’llhavetobeverybriefonthispartbecauseHTMLcanbeveryverbose.
WOW! eBook
www.wowebook.org
Writingthetemplates
Alltemplatesinheritfromabaseone,whichprovidestheHTMLstructureforallothers,
inaveryOOPtypeoffashion.Italsospecifiesafewblocks,whichareareasthatcanbe
overriddenbychildrensothattheycanprovidecustomcontentforthoseareas.Let’sstart
withthebasetemplate:
entries/templates/entries/base.html
{%loadstaticfromstaticfiles%}
<!DOCTYPEhtml>
<htmllang="en">
<head>
{%blockmeta%}
<metacharset="utf-8">
<metaname="viewport"
content="width=device-width,initial-scale=1.0">
{%endblockmeta%}
{%blockstyles%}
<linkhref="{%static"entries/css/main.css"%}"
rel="stylesheet">
{%endblockstyles%}
<title>{%blocktitle%}Title{%endblocktitle%}</title>
</head>
<body>
<divid="page-content">
{%blockpage-content%}
{%endblockpage-content%}
</div>
<divid="footer">
{%blockfooter%}
{%endblockfooter%}
</div>
</body>
</html>
Thereisagoodreasontorepeattheentriesfolderfromthetemplatesone.Whenyou
deployaDjangowebsite,youcollectallthetemplatefilesunderonefolder.Ifyoudon’t
specifythepathslikeIdid,youmaygetabase.htmltemplateintheentriesapp,anda
base.htmltemplateinanotherapp.Thelastonetobecollectedwilloverrideanyother
filewiththesamename.Forthisreason,byputtingtheminatemplates/entriesfolder
andusingthistechniqueforeachDjangoappyouwrite,youavoidtheriskofname
collisions(thesamegoesforanyotherstaticfile).
Thereisnotmuchtosayaboutthistemplate,really,apartfromthefactthatitloadsthe
statictagsothatwecangeteasyaccesstothestaticpathwithouthardcodingitinthe
templatebyusing{%static…%}.Thecodeinthespecial{%...%}sectionsiscodethat
defineslogic.Thecodeinthespecial{{...}}representsvariablesthatwillberendered
onthepage.
WOW! eBook
www.wowebook.org
Wedefinethreeblocks:title,page-content,andfooter,whosepurposeistoholdthe
title,thecontentofthepage,andthefooter.Blockscanbeoptionallyoverriddenbychild
templatesinordertoprovidedifferentcontentwithinthem.
Here’sthefooter:
entries/templates/entries/footer.html
<divclass="footer">
Goback<ahref="{%url"home"%}">home</a>.
</div>
Itgivesusanicelinktothehomepage.
Thehomepagetemplateisthefollowing:
entries/templates/entries/home.html
{%extends"entries/base.html"%}
{%blocktitle%}WelcometotheEntrywebsite.{%endblocktitle%}
{%blockpage-content%}
<h1>Welcome{{user.first_name}}!</h1>
<divclass="home-option">Toseethelistofyourentries
pleaseclick<ahref="{%url"entries"%}">here.</a>
</div>
<divclass="home-option">Toinsertanewentrypleaseclick
<ahref="{%url"insert"%}">here.</a>
</div>
<divclass="home-option">Tologinasanotheruserpleaseclick
<ahref="{%url"logout"%}">here.</a>
</div>
<divclass="home-option">Togototheadminpanel
pleaseclick<ahref="{%url"admin:index"%}">here.</a>
</div>
{%endblockpage-content%}
Itextendsthebase.htmltemplate,andoverridestitleandpage-content.Youcansee
thatbasicallyallitdoesisprovidefourlinkstotheuser.Thesearethelistofentries,the
insertpage,thelogoutpage,andtheadminpage.Allofthisisdonewithouthardcodinga
singleURL,throughtheuseofthe{%url…%}tag,whichisthetemplateequivalentof
thereversefunction.
ThetemplateforinsertinganEntryisasfollows:
entries/templates/entries/insert.html
{%extends"entries/base.html"%}
{%blocktitle%}InsertanewEntry{%endblocktitle%}
{%blockpage-content%}
{%ifmessages%}
{%formessageinmessages%}
<pclass="{{message.tags}}">{{message}}</p>
{%endfor%}
{%endif%}
WOW! eBook
www.wowebook.org
<h1>InsertanewEntry</h1>
<formaction="{%url"insert"%}"method="post">
{%csrf_token%}{{form.as_p}}
<inputtype="submit"value="Insert">
</form><br>
{%endblockpage-content%}
{%blockfooter%}
<div><ahref="{%url"entries"%}">Seeyourentries.</a></div>
{%include"entries/footer.html"%}
{%endblockfooter%}
Thereissomeconditionallogicatthebeginningtodisplaymessages,ifany,andthenwe
definetheform.Djangogivesustheabilitytorenderaformbysimplycalling{{
form.as_p}}(alternatively,form.as_ulorform.as_table).Thiscreatesallthe
necessaryfieldsandlabelsforus.Thedifferencebetweenthethreecommandsisinthe
waytheformislaidout:asaparagraph,asanunorderedlistorasatable.Weonlyneedto
wrapitinformtagsandaddasubmitbutton.Thisbehaviorwasdesignedforour
convenience;weneedthefreedomtoshapethat<form>tagaswewant,soDjangoisn’t
intrusiveonthat.Also,notethat{%csrf_token%}.Itwillberenderedintoatokenby
Djangoandwillbecomepartofthedatasenttotheserveronsubmission.Thisway
Djangowillbeabletoverifythattherequestwasfromanallowedsource,thusavoiding
theaforementionedcross-siterequestforgeryissue.Didyouseehowwehandledthe
tokenwhenwewrotetheviewfortheEntryinsertion?Exactly.Wedidn’twriteasingle
lineofcodeforit.Djangotakescareofitautomaticallythankstoamiddlewareclass
(CsrfViewMiddleware).PleaserefertotheofficialDjangodocumentationtoexplorethis
subjectfurther.
Forthispage,wealsousethefooterblocktodisplayalinktothehomepage.Finally,we
havethelisttemplate,whichisthemostinterestingone.
entries/templates/entries/list.html
{%extends"entries/base.html"%}
{%blocktitle%}Entrieslist{%endblocktitle%}
{%blockpage-content%}
{%ifentries%}
<h1>Yourentries({{entries|length}}found)</h1>
<div><ahref="{%url"insert"%}">Insertnewentry.</a></div>
<tableclass="entries-table">
<thead>
<tr><th>Entry</th><th>Matches</th></tr>
</thead>
<tbody>
{%forentry,matchinentries%}
<trclass="entries-list{%cycle'light-gray''white'%}">
<td>
Pattern:<codeclass="code">
"{{entry.pattern}}"</code><br>
TestString:<codeclass="code">
WOW! eBook
www.wowebook.org
"{{entry.test_string}}"</code><br>
Added:{{entry.date_added}}
</td>
<td>
{%ifmatch%}
Group:{{match.0}}<br>
Subgroups:
{{match.1|default_if_none:"none"}}<br>
GroupDict:{{match.2|default_if_none:"none"}}
{%else%}
Nomatchesfound.
{%endif%}
</td>
</tr>
{%endfor%}
</tbody>
</table>
{%else%}
<h1>Youhavenoentries</h1>
<div><ahref="{%url"insert"%}">Insertnewentry.</a></div>
{%endif%}
{%endblockpage-content%}
{%blockfooter%}
{%include"entries/footer.html"%}
{%endblockfooter%}
Itmaytakeyouawhiletogetusedtothetemplatelanguage,butreally,allthereistoitis
thecreationofatableusingaforloop.Westartbycheckingifthereareanyentriesand,
ifso,wecreateatable.Therearetwocolumns,onefortheEntry,andtheotherforthe
match.
IntheEntrycolumn,wedisplaytheEntryobject(apartfromtheuser)andintheMatches
column,wedisplaythat3-tuplewecreatedintheEntryListView.Notethattoaccessthe
attributesofanobject,weusethesamedotsyntaxweuseinPython,forexample{{
entry.pattern}}or{{entry.test_string}},andsoon.
Whendealingwithlistsandtuples,wecannotaccessitemsusingthesquarebrackets
syntax,soweusethedotoneaswell({{match.0}}isequivalenttomatch[0],andso
on.).Wealsouseafilter,throughthepipe(|)operatortodisplayacustomvalueifa
matchisNone.
TheDjangotemplatelanguage(whichisnotproperlyPython)iskeptsimpleforaprecise
reason.Ifyoufindyourselflimitedbythelanguage,itmeansyou’reprobablytryingtodo
somethinginthetemplatethatshouldactuallybedoneintheview,wherethatlogicis
morerelevant.
Allowmetoshowyouacoupleofscreenshotsofthelistandinserttemplates.Thisis
whatthelistofentrieslookslikeformyfather:
WOW! eBook
www.wowebook.org
Notehowtheuseofthecycletagalternatesthebackgroundcoloroftherowsfromwhite
tolightgray.Thoseclassesaredefinedinthemain.cssfile.
TheEntryinsertionpageissmartenoughtoprovideafewdifferentscenarios.Whenyou
landonitatfirst,itpresentsyouwithjustanemptyform.Ifyoufillitincorrectly,itwill
displayanicemessageforyou(seethefollowingpicture).However,ifyoufailtofillin
bothfields,itwilldisplayanerrormessagebeforethem,alertingyouthatthosefieldsare
required.
Notealsothecustomfooter,whichincludesbothalinktotheentrieslistandalinktothe
homepage:
WOW! eBook
www.wowebook.org
Andthat’sit!YoucanplayaroundwiththeCSSstylesifyouwish.Downloadthecodefor
thebookandhavefunexploringandextendingthisproject.Addsomethingelsetothe
model,createandapplyamigration,playwiththetemplates,there’slotstodo!
Djangoisaverypowerfulframework,andofferssomuchmorethanwhatI’vebeenable
toshowyouinthischapter,soyoudefinitelywanttocheckitout.Thebeautyofitisthat
it’sPython,soreadingitssourcecodeisaveryusefulexercise.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Thefutureofwebdevelopment
Computerscienceisaveryyoungsubject,comparedtootherbranchesofsciencethat
haveexistedalongsidehumankindforcenturiesormore.Oneofitsmaincharacteristicsis
thatitmovesextremelyfast.Itleapsforwardwithsuchspeedthat,injustafewyears,you
canseechangesthatarecomparabletorealworldchangesthattookacenturytohappen.
Therefore,asacoder,youmustpayattentiontowhathappensinthisworld,allthetime.
Somethingthatishappeningnowisthatbecausepowerfulcomputersarenowquitecheap
andalmosteveryonehasaccesstothem,thetrendistotryandavoidputtingtoomuch
workloadonthebackend,andletthefrontendhandlepartofit.Therefore,inthelastfew
years,JavaScriptframeworksandlibrarieslikejQueryandBackbonehavebecomevery
popularandwebdevelopmenthasshiftedfromaparadigmwherethebackendtakescare
ofhandlingdata,preparingit,andservingittothefrontendtodisplayit,toaparadigm
wherethebackendissometimesjustusedasanAPI,asheerdataprovider.Thefrontend
fetchesthedatafromthebackendwithanAPIcall,andthenittakescareoftherest.This
shiftfacilitatestheexistenceofparadigmslikeSingle-PageApplication(SPA),where,
ideally,thewholepageisloadedonceandthenevolves,basedonthecontentthatusually
comesfromthebackend.E-commercewebsitesthatloadtheresultsofasearchinapage
thatdoesn’trefreshthesurroundingstructure,aremadewithsimilartechniques.Browsers
canperformasynchronouscalls(AJAX)thatcanreturndatawhichcanberead,
manipulatedandinjectedbackintothepagewithJavaScriptcode.
So,ifyou’replanningtoworkonwebdevelopment,Istronglysuggestyoutoget
acquaintedwithJavaScript(ifyou’renotalready),andalsowithAPIs.Inthelastfew
pagesofthischapter,I’llgiveyouanexampleofhowtomakeasimpleAPIusingtwo
differentPythonmicroframeworks:FlaskandFalcon.
WOW! eBook
www.wowebook.org
WritingaFlaskview
Flask(http://flask.pocoo.org/)isaPythonmicroframework.Itprovidesfewerfeaturesthan
Django,butit’ssupposedlyfasterandquickertogetupandrunning.Tobehonest,getting
Djangoupandrunningnowadaysisalsoveryquicklydone,butFlaskissopopularthat
it’sgoodtoseeanexampleofit,nonetheless.
Inyourch10folder,createaflaskfolderwiththefollowingstructure:
$tree-Aflask#fromthech10folder
flask
├──main.py
└──templates
└──main.html
Basically,we’regoingtocodetwosimplefiles:aFlaskapplicationandanHTML
template.FlaskusesJinja2astemplateengine.It’sextremelypopularandveryfast,and
justrecentlyevenDjangohasstartedtooffernativesupportforit,whichissomethingthat
Pythoncodershavelongedfor,foralongtime.
flask/templates/main.html
<!doctypehtml>
<title>HellofromFlask</title>
<h1>
{%ifname%}
Hello{{name}}!
{%else%}
Helloshyperson!
{%endif%}
</h1>
Thetemplateisalmostoffensivelysimple;allitdoesistochangethegreetingaccording
tothepresenceofthenamevariable.AbitmoreinterestingistheFlaskapplicationthat
rendersit:
flask/main.py
fromflaskimportFlask,render_template
app=Flask(__name__)
@app.route('/')
@app.route('/<name>')
defhello(name=None):
returnrender_template('main.html',name=name)
if__name__=='__main__':
app.run()
Wecreateanappobject,whichisaFlaskapplication.Weonlyfeedthefully-qualified
nameofthemodule,whichisstoredin__name__.
Then,wewriteasimplehelloview,whichtakesanoptionalnameargument.Inthebody
oftheview,wesimplyrenderthemain.htmltemplate,passingtoitthenameargument,
WOW! eBook
www.wowebook.org
regardlessofitsvalue.
What’sinterestingistherouting.DifferentlyfromDjango’swayoftyingupviewsand
URLs(theurls.pymodule),inFlaskyoudecorateyourviewwithoneormore
@app.routedecorators.Inthiscase,weacceptboththerootURLwithoutanythingelse,
orwithnameinformation.
Changeintotheflaskfolderandtype(makesureyouhaveFlaskinstalledwith$pip
installflask):
$pythonmain.py
Youcanopenabrowserandgotohttp://127.0.0.1:5000/.ThisURLhasnoname
information;therefore,youwillseeHelloshyperson!Itiswrittenallniceandbig.Tryto
addsomethingtothatURLlikehttp://127.0.0.1:5000/Adriano.HitEnterandthepage
willchangetoHelloAdriano!.
Ofcourse,Flaskoffersyoumuchmorethanthisbutwedon’thavetheroomtoseeamore
complexexample.It’sdefinitelyworthexploring,though.Severalprojectsuseit
successfullyandit’sfunanditisnicetocreatewebsitesorAPIswithit.Flask’sauthor,
ArminRonacher,isasuccessfulandveryprolificcoder.Healsocreatedorcollaborated
onseveralotherinterestingprojectslikeWerkzeug,Jinja2,Click,andSphinx.
WOW! eBook
www.wowebook.org
BuildingaJSONquoteserverinFalcon
Falcon(http://falconframework.org/)isanothermicroframeworkwritteninPython,which
wasdesignedtobelight,fastandflexible.Ithinkthisrelativelyyoungprojectwillevolve
tobecomesomethingreallypopularduetoitsspeed,whichisimpressive,soI’mhappyto
showyouatinyexampleusingit.
We’regoingtobuildaviewthatreturnsarandomlychosenquotefromtheBuddha.
Inyourch10folder,createanewonecalledfalcon.We’llhavetwofiles:quotes.pyand
main.py.Torunthisexample,installFalconandGunicorn($pipinstallfalcon
gunicorn).Falconistheframework,andGunicorn(GreenUnicorn)isaPythonWSGI
HTTPServerforUnix(which,inlaymanterms,meansthetechnologythatisusedtorun
theserver).Whenyou’reallsetup,startbycreatingthequotes.pyfile.
falcon/quotes.py
quotes=[
"Thousandsofcandlescanbelightedfromasinglecandle,"
"andthelifeofthecandlewillnotbeshortened."
"Happinessneverdecreasesbybeingshared.",
...
"Peacecomesfromwithin.Donotseekitwithout.",
]
Youwillfindthecompletelistofquotesinthesourcecodeforthisbook.Ifyoudon’thave
it,youcanalsofillinyourfavoritequotes.Notethatnoteverylinehasacommaatthe
end.InPython,it’spossibletoconcatenatestringslikethat,aslongastheyareinbrackets
(orbraces).It’scalledimplicitconcatenation.
Thecodeforthemainappisnotlong,butitisinteresting:
falcon/main.py
importjson
importrandom
importfalcon
fromquotesimportquotes
classQuoteResource:
defon_get(self,req,resp):
quote={
'quote':random.choice(quotes),
'author':'TheBuddha'
}
resp.body=json.dumps(quote)
api=falcon.API()
api.add_route('/quote',QuoteResource())
Let’sstartwiththeclass.InDjangowehadagetmethod,inFlaskwedefinedafunction,
andherewewriteanon_getmethod,anamingstylethatremindsmeofC#event
handlers.Ittakesarequestandaresponseargument,bothautomaticallyfedbythe
framework.Initsbody,wedefineadictwitharandomlychosenquote,andtheauthor
WOW! eBook
www.wowebook.org
information.ThenwedumpthatdicttoaJSONstringandsettheresponsebodytoits
value.Wedon’tneedtoreturnanything,Falconwilltakecareofitforus.
Attheendofthefile,wecreatetheFalconapplication,andwecalladd_routeonittotie
thehandlerwehavejustwrittentotheURLwewant.
Whenyou’reallsetup,changetothefalconfolderandtype:
$gunicornmain:api
Then,makearequest(orsimplyopenthepagewithyourbrowser)to
http://127.0.0.1:8000/quote.WhenIdidit,IgotthisJSONinresponse:
{
quote:"Themindiseverything.Whatyouthinkyoubecome.",
author:"TheBuddha"
}
Whatevertheframeworkyouendupusingforyourwebdevelopment,tryandkeep
yourselfinformedaboutotherchoicestoo.Sometimesyoumaybeinsituationswherea
differentframeworkistherightwaytogo,andhavingaworkingknowledgeofdifferent
toolswillgiveyouanadvantage.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,wecaughtaglimpseofwebdevelopment.Wetalkedaboutimportant
conceptsliketheDRYphilosophyandtheconceptofaframeworkasatoolthatprovides
uswithmanythingsweneedinordertowritecodetoserverequests.Wealsotalkedabout
theMTVpattern,andhownicelythesethreelayersplaytogethertorealizearequestresponsepath.
Lateron,webrieflyintroducedregularexpressions,whichisasubjectofparamount
importance,andit’sthelayerwhichprovidesthetoolsforURLrouting.
Therearemanydifferentframeworksoutthere,andDjangoisdefinitelyoneofthebest
andmostwidelyused,soit’sdefinitelyworthexploring,especiallyitssourcecode,which
isverywellwritten.
Thereareotherveryinterestingandimportantframeworkstoo,likeFlask.Theyprovide
fewerfeaturesbut,ingeneral,theyarefaster,bothinexecutiontimeandtosetup.One
thatisextremelyfastistherelativelyyoungFalconproject,whosebenchmarksare
outstanding.
It’simportanttogetasolidunderstandingofhowtherequest-responsemechanismworks,
andhowtheWebingeneralworks,sothateventuallyitwon’tmattertoomuchwhich
frameworkyouhavetouse.Youwillbeabletopickitupquicklybecauseitwillonlybea
matterofgettingfamiliarwithawayofdoingsomethingyoualreadyknowalotabout.
Exploreatleastthreeframeworksandtrytocomeupwithdifferentusecasestodecide
whichoneofthemcouldbetheidealchoice.Whenyouareabletomakethatchoice,you
willknowyouhaveagoodenoughunderstandingofthem.
Thenextchapterisaboutdebuggingandtroubleshooting.We’lllearnhowtodealwith
errorsandissuessothatifyougetintroublewhencoding(don’tworry,normallyitonly
happensaboutallthetime),youwillbeabletoquicklyfindthesolutiontoyourproblem
andmoveon.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter11.Debuggingand
Troubleshooting
“Ifdebuggingistheprocessofremovingsoftwarebugs,thenprogrammingmustbetheprocessofputtingthemin.” —EdsgerW.Dijkstra
Inthelifeofaprofessionalcoder,debuggingandtroubleshootingtakeupasignificant
amountoftime.Evenifyouworkonthemostbeautifulcodebaseeverwrittenbyman,
therewillstillbebugsinit,thatisguaranteed.
Wespendanawfullotoftimereadingotherpeople’scodeand,inmyopinion,agood
softwaredeveloperissomeonewhokeepstheirattentionhigh,evenwhenthey’rereading
codethatisnotreportedtobewrongorbuggy.
Beingabletodebugcodeefficientlyandquicklyisaskillthatanycoderneedstokeep
improving.Somethinkthatbecausetheyhavereadthemanual,they’refine,butthereality
is,thenumberofvariablesinthegameissobigthatthereisnomanual.Thereare
guidelinesthatonecanfollow,butthereisnomagicbookthatwillteachyoueverything
youneedtoknowinordertobecomegoodatthis.
Ifeelthatonthisparticularsubject,Ihavelearnedthemostfrommycolleagues.Itamazes
metoobservesomeoneveryskilledattackingaproblem.Ienjoyseeingthestepsthey
take,thethingstheyverifytoexcludepossiblecauses,andthewaytheyconsiderthe
suspectsthateventuallyleadthemtothesolutiontotheproblem.
Everycolleagueweworkwithcanteachussomething,orsurpriseuswithafantastic
guessthatturnsouttobetherightone.Whenthathappens,don’tjustremainin
wonderment(orworse,inenvy),butseizethemomentandaskthemhowtheygottothat
guessandwhy.Theanswerwillallowyoutoseeifthereissomethingyoucanstudyin
deeplateronsothat,maybenexttime,you’llbetheonewhowillcatchthebug.
Somebugsareveryeasytospot.Theycomeoutofcoarsemistakesand,onceyouseethe
effectsofthosemistakes,it’seasytofindasolutionthatfixestheproblem.
Butthereareotherbugswhicharemuchmoresubtle,muchmoreslippery,andrequire
trueexpertise,andagreatdealofcreativityandout-of-the-boxthinking,tobedealtwith.
Theworstofall,atleastforme,arethenondeterministicones.Thesesometimeshappen,
andsometimesdon’t.SomehappenonlyinenvironmentAbutnotinenvironmentB,even
thoughAandBaresupposedtobeexactlythesame.Thosebugsarethetrueevilones,
andtheycandriveyoucrazy.
Andofcourse,bugsdon’tjusthappeninthesandbox,right?Withyourbosstellingyou
“don’tworry!takeyourtimetofixthis,havelunchfirst!“.Nope.TheyhappenonaFriday
athalfpastfive,whenyourbrainiscookedandyoujustwanttogohome.It’sinthose
moments,wheneveryoneisgettingupsetinasplitsecond,whenyourbossisbreathingon
yourneck,thatyouhavetobeabletokeepcalm.AndIdomeanit.That’sthemost
WOW! eBook
www.wowebook.org
importantskilltohaveifyouwanttobeabletofightbugseffectively.Ifyouallowyour
mindtogetstressed,saygoodbyetocreativethinking,tologicdeduction,andto
everythingyouneedatthatmoment.Sotakeadeepbreath,sitproperly,andfocus.
Inthischapter,Iwilltrytodemonstratesomeusefultechniquesthatyoucanemploy
accordingtotheseverityofthebug,andafewsuggestionsthatwillhopefullyboostyour
weaponsagainstbugsandissues.
WOW! eBook
www.wowebook.org
Debuggingtechniques
Inthispart,I’llpresentyouwiththemostcommontechniques,theonesIusemostoften,
however,pleasedon’tconsiderthislisttobeexhaustive.
WOW! eBook
www.wowebook.org
Debuggingwithprint
Thisisprobablytheeasiesttechniqueofall.It’snotveryeffective,itcannotbeused
everywhereanditrequiresaccesstoboththesourcecodeandaterminalthatwillrunit
(andthereforeshowtheresultsoftheprintfunctioncalls).
However,inmanysituations,thisisstillaquickandusefulwaytodebug.Forexample,if
youaredevelopingaDjangowebsiteandwhathappensinapageisnotwhatwouldyou
expect,youcanfilltheviewwithprintsandkeepaneyeontheconsolewhileyoureload
thepage.I’veprobablydoneitamilliontimes.
Whenyouscattercallstoprintinyourcode,younormallyendupinasituationwhere
youduplicatealotofdebuggingcode,eitherbecauseyou’reprintingatimestamp(likewe
didwhenweweremeasuringhowfastlistcomprehensionsandgeneratorswere),or
becauseyouhavetosomehowbuildastringofsomesortthatyouwanttodisplay.
Anotherissueisthatit’sextremelyeasytoforgetcallstoprintinyourcode.
So,forthesereasons,ratherthanusingabarecalltoprint,Isometimesprefertocodea
customfunction.Let’sseehow.
WOW! eBook
www.wowebook.org
Debuggingwithacustomfunction
Havingacustomfunctioninasnippetthatyoucanquicklygrabandpasteintothecode,
andthenusetodebug,canbeveryuseful.Ifyou’refast,youcanalwayscodeoneonthe
fly.Theimportantthingistocodeitinawaythatitwon’tleavestuffaroundwhenyou
eventuallyremovethecallsanditsdefinition,thereforeit’simportanttocodeitinaway
thatiscompletelyself-contained.Anothergoodreasonforthisrequirementisthatitwill
avoidpotentialnameclasheswiththerestofthecode.
Let’sseeanexampleofsuchafunction.
custom.py
defdebug(*msg,print_separator=True):
print(*msg)
ifprint_separator:
print('-'*40)
debug('Datais…')
debug('Different','Strings','Arenotaproblem')
debug('Afterwhileloop',print_separator=False)
Inthiscase,Iamusingakeyword-onlyargumenttobeabletoprintaseparator,whichisa
lineof40dashes.
Thefunctionisverysimple,Ijustredirectwhateverisinmsgtoacalltoprintand,if
print_separatorisTrue,Iprintalineseparator.Runningthecodewillshow:
$pythoncustom.py
Datais…
---------------------------------------DifferentStringsArenotaproblem
---------------------------------------Afterwhileloop
Asyoucansee,thereisnoseparatorafterthelastline.
Thisisjustoneeasywaytosomehowaugmentasimplecalltotheprintfunction.Let’s
seehowwecancalculateatimedifferencebetweencalls,usingoneofPython’stricky
featurestoouradvantage.
custom_timestamp.py
fromtimeimportsleep
defdebug(*msg,timestamp=[None]):
print(*msg)
fromtimeimporttime#localimport
iftimestamp[0]isNone:
timestamp[0]=time()#1
else:
now=time()
print('Timeelapsed:{:.3f}s'.format(
now-timestamp[0]))
timestamp[0]=now#2
WOW! eBook
www.wowebook.org
debug('Enteringnastypieceofcode…')
sleep(.3)
debug('Firststepdone.')
sleep(.5)
debug('Secondstepdone.')
Thisisabittrickier,butstillquitesimple.Firstnoticeweimportthetimefunctionfrom
thetimemodulefromthedebugfunction.Thisallowsustoavoidhavingtoaddthat
importoutsideofthefunction,andmaybeforgetitthere.
TakealookathowIdefinedtimestamp.It’salist,ofcourse,butwhat’simportanthereis
thatitisamutableobject.ThismeansthatitwillbesetupwhenPythonparsesthe
functionanditwillretainitsvaluethroughoutdifferentcalls.Therefore,ifweputa
timestampinitaftereachcall,wecankeeptrackoftimewithouthavingtouseanexternal
globalvariable.Iborrowedthistrickfrommystudiesonclosures,atechniquethatI
encourageyoutoreadaboutbecauseit’sveryinteresting.
Right,so,afterhavingprintedwhatevermessagewehadtoprintandimportingtime,we
theninspectthecontentoftheonlyitemintimestamp.IfitisNone,wehavenoprevious
reference,thereforewesetthevaluetothecurrenttime(#1).
Ontheotherhand,ifwehaveapreviousreference,wecancalculateadifference(which
wenicelyformattothreedecimaldigits)andthenwefinallyputthecurrenttimeagainin
timestamp(#2).It’sanicetrick,isn’tit?
Runningthiscodeshowsthisresult:
$pythoncustom_timestamp.py
Enteringnastypieceofcode…
Firststepdone.
Timeelapsed:0.300s
Secondstepdone.
Timeelapsed:0.501s
Whateverisyoursituation,havingaselfcontainedfunctionlikethiscanbeveryuseful.
WOW! eBook
www.wowebook.org
Inspectingthetraceback
WebrieflytalkedaboutthetracebackinChapter7,Testing,Profiling,andDealingwith
Exceptionswhenwesawseveraldifferentkindsofexceptions.Thetracebackgivesyou
informationaboutwhathappenedinyourapplicationthatwentwrong.Yougetagreat
helpfromreadingit.Let’sseeaverysmallexample:
traceback_simple.py
d={'some':'key'}
key='some-other'
print(d[key])
Wehaveadictandwehavetriedtoaccessakeywhichisn’tinit.Youshouldremember
thatthiswillraiseaKeyErrorexception.Let’srunthecode:
$pythontraceback_simple.py
Traceback(mostrecentcalllast):
File"traceback_simple.py",line3,in<module>
print(d[key])
KeyError:'some-other'
Youcanseethatwegetalltheinformationweneed:themodulename,thelinethatcaused
theerror(boththenumberandtheinstruction),andtheerroritself.Withthisinformation,
youcangobacktothesourcecodeandtryandunderstandwhat’sgoingwrong.
Let’snowcreateamoreinterestingexamplethatbuildsonthis,andexercisesafeature
thatisonlyavailableinPython3.Imaginethatwe’revalidatingadict,workingon
mandatoryfields,thereforeweexpectthemtobethere.Ifnot,weneedtoraiseacustom
ValidationError,thatwewilltrapfurtherupstreamintheprocessthatrunsthevalidator
(whichisnotshownhere,itcouldbeanything,really).Itshouldbesomethinglikethis:
traceback_validator.py
classValidatorError(Exception):
"""RaisedwhenaccessingadictresultsinKeyError."""
d={'some':'key'}
mandatory_key='some-other'
try:
print(d[mandatory_key])
exceptKeyError:
raiseValidatorError(
'`{}`notfoundind.'.format(mandatory_key))
Wedefineacustomexceptionthatisraisedwhenthemandatorykeyisn’tthere.Notethat
itsbodyconsistsofitsdocumentationstringsowedon’tneedtoaddanyotherstatements.
Verysimply,wedefineadummydictandtrytoaccessitusingmandatory_key.Wetrap
theKeyErrorandraiseValidatorErrorwhenthathappens.Thepurposeofdoingthisis
thatwemayalsowanttoraiseValidatorErrorinothercircumstances,notnecessarilyas
aconsequenceofamandatorykeybeingmissing.Thistechniqueallowsustorunthe
validationinasimpletry/exceptthatonlycaresaboutValidatorError.
WOW! eBook
www.wowebook.org
Thethingis,inPython2,thiscodewouldjustdisplaythelastexception
(ValidatorError),whichmeanswewouldlosetheinformationabouttheKeyErrorthat
precedesit.InPython3,thisbehaviorhaschangedandexceptionsarenowchainedsothat
youhaveamuchbetterinformationreportwhensomethinghappens.Thecodeproduces
thisresult:
$pythontraceback_validator.py
Traceback(mostrecentcalllast):
File"traceback_validator.py",line7,in<module>
print(d[mandatory_key])
KeyError:'some-other'
Duringhandlingoftheaboveexception,anotherexceptionoccurred:
Traceback(mostrecentcalllast):
File"traceback_validator.py",line10,in<module>
'`{}`notfoundind.'.format(mandatory_key))
__main__.ValidatorError:`some-other`notfoundind.
Thisisbrilliant,becausewecanseethetracebackoftheexceptionthatledustoraise
ValidationError,aswellasthetracebackfortheValidationErroritself.
Ihadanicediscussionwithoneofmyreviewersaboutthetracebackyougetfromthepip
installer.Hewashavingtroublesettingeverythingupinordertoreviewthecodefor
Chapter9,DataScience.HisfreshUbuntuinstallationwasmissingafewlibrariesthat
wereneededbythepippackagesinordertoruncorrectly.
Thereasonhewasblockedwasthathewastryingtofixtheerrorsdisplayedinthe
tracebackstartingfromthetopone.Isuggestedthathestartedfromthebottomone
instead,andfixthat.Thereasonwasthat,iftheinstallerhadgottentothatlastline,Iguess
thatbeforethat,whatevererrormayhaveoccurred,itwasstillpossibletorecoverfromit.
Onlyafterthelastline,pipdecideditwasn’tpossibletocontinueanyfurther,and
thereforeIstartedfixingthatone.Oncethelibrariesrequiredtofixthaterrorhadbeen
installed,everythingelsewentsmoothly.
Readingatracebackcanbetricky,andmyfriendwaslackingthenecessaryexperienceto
addressthisproblemcorrectly,therefore,ifyouendupinthesamesituation,don’tbe
discouraged,andtrytoshakethingsupabit,don’ttakeanythingforgranted.
Pythonhasahugeandwonderfulcommunityandit’sveryunlikelythat,whenyou
encounteraproblem,you’rethefirstonetoseeit,soopenabrowserandsearch.Bydoing
so,yoursearchingskillswillalsoimprovebecauseyouwillhavetotrimtheerrordownto
theminimumbutessentialsetofdetailsthatwillmakeyoursearcheffective.
Ifyouwanttoplayandunderstandthetracebackabitbetter,inthestandardlibrarythere
isamodulecalled,surprisesurprise,tracebackthatyoucanuse.Itprovidesastandard
interfacetoextract,format,andprintstacktracesofPythonprograms,mimickingexactly
thebehaviorofthePythoninterpreterwhenitprintsastacktrace.
WOW! eBook
www.wowebook.org
UsingthePythondebugger
AnotherveryeffectivewayofdebuggingPythonistousethePythondebugger:pdb.If
youareaddictedtotheIPythonconsole,likeme,youshoulddefinitelycheckouttheipdb
library.ipdbaugmentsthestandardpdbinterfacelikeIPythondoeswiththePython
console.
Thereareseveraldifferentwaysofusingthisdebugger(whicheverversion,itisnot
important),butthemostcommononeconsistsofsimplysettingabreakpointandrunning
thecode.WhenPythonreachesthebreakpoint,executionissuspendedandyouget
consoleaccesstothatpointsothatyoucaninspectallthenames,andsoon.Youcanalso
alterdataontheflytochangetheflowoftheprogram.
Asatoyexample,let’spretendwehaveaparserthatisraisingaKeyErrorbecauseakey
ismissinginadict.ThedictisfromaJSONpayloadthatwecannotcontrol,andwejust
want,forthetimebeing,tocheatandpassthatcontrol,sincewe’reinterestedinwhat
comesafterwards.Let’sseehowwecouldinterceptthismoment,inspectthedata,fixit
andgettothebottom,withipdb.
ipdebugger.py
#dcomesfromaJSONpayloadwedon'tcontrol
d={'first':'v1','second':'v2','fourth':'v4'}
#keysalsocomesfromaJSONpayloadwedon'tcontrol
keys=('first','second','third','fourth')
defdo_something_with_value(value):
print(value)
forkeyinkeys:
do_something_with_value(d[key])
print('Validationdone.')
Asyoucansee,thiscodewillbreakwhenkeygetsthevalue'third',whichismissingin
thedict.Remember,we’repretendingthatbothdandkeyscomedynamicallyfroma
JSONpayloadwedon’tcontrol,soweneedtoinspecttheminordertofixdandpassthe
forloop.Ifwerunthecodeasitis,wegetthefollowing:
$pythonipdebugger.py
v1
v2
Traceback(mostrecentcalllast):
File"ipdebugger.py",line10,in<module>
do_something_with_value(d[key])
KeyError:'third'
Soweseethatthatkeyismissingfromthedict,butsinceeverytimewerunthiscodewe
maygetadifferentdictorkeystuple,thisinformationdoesn’treallyhelpus.Let’sinjecta
calltoipdb.
ipdebugger_ipdb.py
WOW! eBook
www.wowebook.org
#dcomesfromaJSONpayloadwedon'tcontrol
d={'first':'v1','second':'v2','fourth':'v4'}
#keysalsocomesfromaJSONpayloadwedon'tcontrol
keys=('first','second','third','fourth')
defdo_something_with_value(value):
print(value)
importipdb
ipdb.set_trace()#weplaceabreakpointhere
forkeyinkeys:
do_something_with_value(d[key])
print('Validationdone.')
Ifwenowrunthiscode,thingsgetinteresting(notethatyouroutputmayvaryalittleand
thatallthecommentsinthisoutputwereaddedbyme):
$pythonipdebugger_ipdb.py
>/home/fab/srv/l.p/ch11/ipdebugger_ipdb.py(12)<module>()
11
--->12forkeyinkeys:#thisiswherethebreakpointcomes
13do_something_with_value(d[key])
ipdb>keys#let'sinspectthekeystuple
('first','second','third','fourth')
ipdb>!d.keys()#nowthekeysofd
dict_keys(['first','fourth','second'])#wemiss'third'
ipdb>!d['third']='somethingdarkside…'#let'sputitin
ipdb>c#...andcontinue
v1
v2
somethingdarkside…
v4
Validationdone.
Thisisveryinteresting.First,notethat,whenyoureachabreakpoint,you’reserveda
consolethattellsyouwhereyouare(thePythonmodule)andwhichlineisthenextoneto
beexecuted.Youcan,atthispoint,performabunchofexploratoryactions,suchas
inspectingthecodebeforeandafterthenextline,printingastacktrace,interactingwith
theobjects,andsoon.PleaseconsulttheofficialPythondocumentationonpdbtolearn
moreaboutthis.Inourcase,wefirstinspectthekeystuple.Afterthat,weinspectthekeys
ofd.
HaveyounoticedthatexclamationmarkIprependedtod?It’sneededbecausedisa
commandinthepdbinterfacethatmovestheframe(d)own.
Note
Iindicatecommandswithintheipdbshellwiththisnotation:eachcommandisactivated
byoneletter,whichtypicallyisthefirstletterofthecommandname.So,dfordown,nfor
next,andsforstepbecome,moreconcisely,(d)own,(n)extand(s)tep.
Iguessthisisagoodenoughreasontohavebetternames,right?Indeed,butIneededto
WOW! eBook
www.wowebook.org
showyouthis,soIchosetoused.Inordertotellpdbthatwe’renotyieldinga(d)own
command,weput“!”infrontofdandwe’refine.
Afterseeingthekeysofd,weseethat'third'ismissing,soweputitinourselves(could
thisbedangerous?thinkaboutit).Finally,nowthatallthekeysarein,wetypec,which
means(c)ontinue.
pdbalsogivesyoutheabilitytoproceedwithyourcodeonelineatatimeusing(n)ext,to
(s)tepintoafunctionfordeeperanalysis,orhandlingbreakswith(b)reak.Foracomplete
listofcommands,pleaserefertothedocumentationortype(h)elpintheconsole.
Youcanseefromtheoutputthatwecouldfinallygettotheendofthevalidation.
pdb(oripdb)areinvaluabletoolsthatIuseeveryday,Icouldn’tlivewithoutthem.So,go
andhavefun,setabreakpointsomewhereandtryandinspect,followtheofficial
documentationandtrythecommandsinyourcodetoseetheireffectandlearnthemwell.
WOW! eBook
www.wowebook.org
Inspectinglogfiles
Anotherwayofdebuggingamisbehavingapplicationistoinspectitslogfiles.Logfiles
arespecialfilesinwhichanapplicationwritesdownallsortsofthings,normallyrelatedto
what’sgoingoninsideofit.Ifanimportantprocedureisstarted,Iwouldtypicallyexpect
alineforthatinthelogs.Itisthesamewhenitfinishes,andpossiblyforwhathappens
insideofit.
Errorsneedtobeloggedsothatwhenaproblemhappenswecaninspectwhatwentwrong
bytakingalookattheinformationinthelogfiles.
TherearemanydifferentwaystosetupaloggerinPython.Loggingisverymalleableand
youcanconfigureit.Inanutshell,therearenormallyfourplayersinthegame:loggers,
handlers,filters,andformatters:
Loggersexposetheinterfacethattheapplicationcodeusesdirectly
Handlerssendthelogrecords(createdbyloggers)totheappropriatedestination
Filtersprovideafinergrainedfacilityfordeterminingwhichlogrecordstooutput
Formattersspecifythelayoutofthelogrecordsinthefinaloutput
LoggingisperformedbycallingmethodsoninstancesoftheLoggerclass.Eachlineyou
loghasalevel.Thelevelsnormallyusedare:DEBUG,INFO,WARNING,ERROR,andCRITICAL.
Youcanimportthemfromtheloggingmodule.Theyareinorderofseverityandit’svery
importanttousethemproperlybecausetheywillhelpyoufilterthecontentsofalogfile
basedonwhatyou’researchingfor.Logfilesusuallybecomeextremelybigsoit’svery
importanttohavetheinformationinthemwrittenproperlysothatyoucanfinditquickly
whenitmatters.
Youcanlogtoafilebutyoucanalsologtoanetworklocation,toaqueue,toaconsole,
andsoon.Ingeneral,ifyouhaveanarchitecturethatisdeployedononemachine,logging
toafileisacceptable,butwhenyourarchitecturespansovermultiplemachines(suchasin
thecaseofservice-orientedarchitectures),it’sveryusefultoimplementacentralized
solutionforloggingsothatalllogmessagescomingfromeachservicecanbestoredand
investigatedinasingleplace.Ithelpsalot,otherwiseyoucanreallygocrazytryingto
correlategiantfilesfromseveraldifferentsourcestofigureoutwhatwentwrong.
Note
Aservice-orientedarchitecture(SOA)isanarchitecturalpatterninsoftwaredesignin
whichapplicationcomponentsprovideservicestoothercomponentsviaa
communicationsprotocol,typicallyoveranetwork.Thebeautyofthissystemisthat,
whencodedproperly,eachservicecanbewritteninthemostappropriatelanguageto
serveitspurpose.Theonlythingthatmattersisthecommunicationwiththeother
services,whichneedstohappenviaacommonformatsothatdataexchangecanbedone.
Here,Iwillpresentyouwithaverysimpleloggingexample.Wewilllogafewmessages
toafile:
log.py
WOW! eBook
www.wowebook.org
importlogging
logging.basicConfig(
filename='ch11.log',
level=logging.DEBUG,#minimumlevelcaptureinthefile
format='[%(asctime)s]%(levelname)s:%(message)s',
datefmt='%m/%d/%Y%I:%M:%S%p')
mylist=[1,2,3]
logging.info('Startingtoprocess`mylist`...')
forpositioninrange(4):
try:
logging.debug('Valueatposition{}is{}'.format(
position,mylist[position]))
exceptIndexError:
logging.exception('Faultyposition:{}'.format(position))
logging.info('Doneparsing`mylist`.')
Let’sgothroughitlinebyline.First,weimporttheloggingmodule,thenwesetupa
basicconfiguration.Ingeneral,aproductionloggingconfigurationismuchmore
complicatedthanthis,butIwantedtokeepthingsaseasyaspossible.Wespecifya
filename,theminimumlogginglevelwewanttocaptureinthefile,andthemessage
format.We’lllogthedateandtimeinformation,thelevel,andthemessage.
Iwillstartbylogginganinfomessagethattellsmewe’reabouttoprocessourlist.Then,
Iwilllog(thistimeusingtheDEBUGlevel,byusingthedebugfunction)whichisthevalue
atsomeposition.I’musingdebugherebecauseIwanttobeabletofilterouttheselogsin
thefuture(bysettingtheminimumleveltologging.INFOormore),becauseImighthave
tohandleverybiglistsandIdon’twanttologallthevalues.
IfwegetanIndexError(andwedo,sinceI’mloopingoverrange(4)),wecall
logging.exception(),whichisthesameaslogging.error(),butitalsoprintsthe
traceback.
Attheendofthecode,Iloganotherinfomessagesayingwe’redone.Theresultisthis:
[10/08/201504:17:06PM]INFO:Startingtoprocess`mylist`...
[10/08/201504:17:06PM]DEBUG:Valueatposition0is1
[10/08/201504:17:06PM]DEBUG:Valueatposition1is2
[10/08/201504:17:06PM]DEBUG:Valueatposition2is3
[10/08/201504:17:06PM]ERROR:Faultyposition:3
Traceback(mostrecentcalllast):
File"log.py",line15,in<module>
position,mylist[position]))
IndexError:listindexoutofrange
[10/08/201504:17:06PM]INFO:Doneparsing`mylist`.
Thisisexactlywhatweneedtobeabletodebuganapplicationthatisrunningonabox,
andnotonourconsole.Wecanseewhatwenton,thetracebackofanyexceptionraised,
andsoon.
Note
WOW! eBook
www.wowebook.org
Theexamplepresentedhereonlyscratchesthesurfaceoflogging.Foramorein-depth
explanation,youcanfindaveryniceintroductioninthehowto
(https://docs.python.org/3.4/howto/logging.html)sectionoftheofficialPython
documentation.
Loggingisanart,youneedtofindagoodbalancebetweenloggingeverythingand
loggingnothing.Ideally,youshouldloganythingthatyouneedtomakesureyour
applicationisworkingcorrectly,andpossiblyallerrorsorexceptions.
WOW! eBook
www.wowebook.org
Othertechniques
Inthisfinalsection,I’dliketodemonstratebrieflyacoupleoftechniquesthatyoumay
finduseful.
Profiling
WetalkedaboutprofilinginChapter7,Testing,Profiling,andDealingwithExceptions,
andI’monlymentioningitherebecauseprofilingcansometimesexplainweirderrorsthat
areduetoacomponentbeingtooslow.Especiallywhennetworkingisinvolved,having
anideaofthetimingsandlatenciesyourapplicationhastogothroughisveryimportantin
ordertounderstandwhatmaybegoingonwhenproblemsarise,thereforeIsuggestyou
getacquaintedwithprofilingtechniquesalsoforatroubleshootingperspective.
Assertions
Assertionsareanicewaytomakeyourcodeensureyourassumptionsareverified.Ifthey
are,allproceedsregularlybut,iftheyarenot,yougetaniceexceptionthatyoucanwork
with.Sometimes,insteadofinspecting,it’squickertodropacoupleofassertionsinthe
codejusttoexcludepossibilities.Let’sseeanexample:
assertions.py
mylist=[1,2,3]#thisideallycomesfromsomeplace
assert4==len(mylist)#thiswillbreak
forpositioninrange(4):
print(mylist[position])
Thiscodesimulatesasituationinwhichmylistisn’tdefinedbyuslikethat,ofcourse,but
we’reassumingithasfourelements.Soweputanassertionthere,andtheresultisthis:
$pythonassertions.py
Traceback(mostrecentcalllast):
File"assertions.py",line3,in<module>
assert4==len(mylist)
AssertionError
Thistellsusexactlywheretheproblemis.
WOW! eBook
www.wowebook.org
Wheretofindinformation
InthePythonofficialdocumentation,thereisasectiondedicatedtodebuggingand
profiling,whereyoucanreadupaboutthebdbdebuggerframework,andaboutmodules
suchasfaulthandler,timeit,trace,tracemallock,andofcoursepdb.Justheadtothe
standardlibrarysectioninthedocumentationandyou’llfindallthisinformationvery
easily.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Troubleshootingguidelines
Inthisshortsection,I’llliketogiveyouafewtipsthatcomefrommytroubleshooting
experience.
WOW! eBook
www.wowebook.org
Usingconsoleeditors
First,getcomfortableusingvimornanoasaneditor,andlearnthebasicsoftheconsole.
Whenthingsbreakbadyoudon’thavetheluxuryofyoureditorwithallthebellsand
whistlesthere.Youhavetoconnecttoaboxandworkfromthere.Soit’saverygoodidea
tobecomfortablebrowsingyourproductionenvironmentwithconsolecommands,andbe
abletoeditfilesusingconsole-basededitorssuchasvi,vim,ornano.Don’tletyourusual
developmentenvironmentspoilyou,becauseyou’llhavetopayapriceifyoudo.
WOW! eBook
www.wowebook.org
Wheretoinspect
Mysecondsuggestionisonwheretoplaceyourdebuggingbreakpoints.Itdoesn’tmatter
ifyouareusingprint,acustomfunction,oripdb,youstillhavetochoosewheretoplace
thecallsthatprovideyouwiththeinformation,right?
Well,someplacesarebetterthanothers,andtherearewaystohandlethedebugging
progressionthatarebetterthanothers.
Inormallyavoidplacingabreakpointinanifclausebecause,ifthatclauseisnot
exercised,IlosethechanceofgettingtheinformationIwanted.Sometimesit’snoteasyor
quicktogettothebreakpoint,sothinkcarefullybeforeplacingthem.
Anotherimportantthingiswheretostart.Imaginethatyouhave100linesofcodethat
handleyourdata.Datacomesinatline1,andsomehowit’swrongatline100.Youdon’t
knowwherethebugis,sowhatdoyoudo?Youcanplaceabreakpointatline1and
patientlygothroughallthelines,checkingyourdata.Intheworstcasescenario,99lines
later(andmanycoffeecups)youspotthebug.So,considerusingadifferentapproach.
Youstartatline50,andinspect.Ifthedataisgood,itmeansthebughappenslater,in
whichcaseyouplaceyournextbreakpointatline75.Ifthedataatline50isalreadybad,
yougoonbyplacingabreakpointatline25.Then,yourepeat.Eachtime,youmove
eitherbackwardsorforwards,byhalfthejumpyoudidlasttime.
Inourworstcasescenario,yourdebuggingwouldgofrom1,2,3,…,99to50,75,87,93,
96,…,99whichiswayfaster.Infact,it’slogarithmic.Thissearchingtechniqueiscalled
binarysearch,it’sbasedonadivideandconquerapproachandit’sveryeffective,sotry
tomasterit.
WOW! eBook
www.wowebook.org
Usingteststodebug
DoyourememberChapter7,Testing,Profiling,andDealingwithExceptions,abouttests?
Well,ifwehaveabugandalltestsarepassing,itmeanssomethingiswrongormissingin
ourtestcodebase.So,oneapproachistomodifythetestsinsuchawaythattheycaterfor
thenewedgecasethathasbeenspotted,andthenworkyourwaythroughthecode.This
approachcanbeverybeneficial,becauseitmakessurethatyourbugwillbecoveredbya
testwhenit’sfixed.
WOW! eBook
www.wowebook.org
Monitoring
Monitoringisalsoveryimportant.Softwareapplicationscangocompletelycrazyand
havenon-deterministichiccupswhentheyencounteredgecasesituationssuchasthe
networkbeingdown,aqueuebeingfull,anexternalcomponentbeingunresponsive,and
soon.Inthesecases,it’simportanttohaveanideaofwhatwasthebigpicturewhenthe
problemhappenedandbeabletocorrelateittosomethingrelatedtoitinasubtle,perhaps
mysteriousway.
YoucanmonitorAPIendpoints,processes,webpagesavailabilityandloadtime,and
basicallyalmosteverythingthatyoucancode.Ingeneral,whenstartinganapplication
fromscratch,itcanbeveryusefultodesignitkeepinginmindhowyouwanttomonitor
it.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthisshortchapter,wesawdifferenttechniquesandsuggestionstodebugand
troubleshootourcode.Debuggingisanactivitythatisalwayspartofasoftware
developer’swork,soit’simportanttobegoodatit.
Ifapproachedwiththecorrectattitude,itcanbefunandrewarding.
Wesawtechniquestoinspectourcodebaseonfunctions,logging,debuggers,traceback
information,profiling,andassertions.Wesawsimpleexamplesofmostofthemandwe
alsotalkedaboutasetofguidelinesthatwillhelpwhenitcomestofacethefire.
Justremembertoalwaysstaycalmandfocused,anddebuggingwillbeeasieralready.
Thistoo,isaskillthatneedstobelearnedandit’sthemostimportant.Anagitatedand
stressedmindcannotworkproperly,logicallyandcreatively,therefore,ifyoudon’t
strengthenit,itwillbehardforyoutoputallofyourknowledgetogooduse.
Inthenextchapter,wewillendthebookwithanothersmallprojectwhosegoalistoleave
youmorethirstythanyouwerewhenyoustartedthisjourneywithme.
Ready?
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Chapter12.SummingUp–AComplete
Example
“Donotdwellinthepast,donotdreamofthefuture,concentratethemindonthepresentmoment.” —TheShakyamuniBuddha
Inthischapter,Iwillshowyouonelastproject.Ifyou’veworkedwellintherestofthe
book,thisexampleshouldbeeasy.Itriedmybesttocraftitinawaythatitwillneitherbe
toohardforthosewhohaveonlyreadthebook,nortoosimpleforthosewhoalsotookthe
timetoworkontheexamples,andmaybehavereaduponthelinksandtopicsIsuggested.
WOW! eBook
www.wowebook.org
Thechallenge
Oneproblemthatweallhavethesedaysisrememberingpasswords.Wehavepasswords
foreverything:websites,phones,cards,bankaccounts,andsoon.Theamountof
informationwehavetomemorizeisjusttoomuch,somanypeopleendupusingthesame
passwordoverandoveragain.Thisisverybad,ofcourse,soatsomepoint,toolswere
inventedtoalleviatethisproblem.OneofthesetoolsiscalledKeepassX,andbasicallyit
workslikethis:youstartthesoftwarebysettingupaspecialpasswordcalledmaster
password.Onceinside,youstorearecordforeachpasswordyouneedtomemorize,for
example,youre-mailaccount,thebankwebsite,creditcardinformation,andsoon.When
youclosethesoftware,itencryptsthedatabaseusedtostoreallthatinformation,sothat
thedatacanonlybeaccessedbytheownerofthemasterpassword.Therefore,kindofina
LordofTheRingsfashion,byjustowningonepassword,yourulethemall.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Ourimplementation
Ourgoalinthischapteristocreatesomethingsimilarbutweb-based,andthewayIwant
toimplementitisbywritingtwoapplications.
OnewillbeanAPIwritteninFalcon.Itspurposewillbetwofold,itwillbeabletoboth
generateandvalidatepasswords.Itwillprovidethecallerwithinformationaboutthe
validityandascorewhichshouldindicatehowstrongthepasswordis.
ThesecondapplicationisaDjangowebsite,whichwillprovidetheinterfacetohandle
records.Eachrecordwillretaininformationsuchastheusername,e-mail,password,URL,
andsoon.Itwillshowalistofalltherecords,anditwillallowtheusertocreate,update
anddeletethem.Passwordswillbeencryptedbeforebeingstoredinthedatabase.
Thepurposeofthewholeprojectis,therefore,tomimicthewayKeepassXworks,even
thoughitisinamuchsimplerfashion.Itwillbeuptoyou,ifyoulikethisidea,todevelop
itfurtherinordertoaddotherfeaturesandmakeitmoresecure.Iwillmakesuretogive
yousomesuggestionsonhowtoextendit,towardstheend.
Thischapterwillthereforebequitedense,code-wise.It’sthepriceIhavetopayforgiving
youaninterestingexampleinarestrictedamountofspace.
Beforewestart,pleasemakesureyouarecomfortablewiththeprojectspresentedin
Chapter10,WebDevelopmentDoneRightsothatyou’refamiliarwiththebasicsofweb
development.Makesurealsothatyouhaveinstalledallthepippackagesneededforthis
project:django,falcon,cryptography,andnose-parameterized.Ifyoudownloadthe
sourcecodeforthebook,you’llfindeverythingyouneedtoinstallintherequirements
folder,whilethecodeforthischapterwillbeinch12.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
ImplementingtheDjangointerface
Ihopeyou’recomfortablewiththeconceptspresentedinChapter10,WebDevelopment
DoneRightwhichwasmostlyaboutDjango.Ifyouhaven’treadit,thisisprobablyagood
time,beforereadingonhere.
WOW! eBook
www.wowebook.org
Thesetup
Inyourrootfolder(ch12,forme),whichwillcontaintherootfortheinterfaceandtheroot
fortheAPI,startbyrunningthiscommand:
$django-adminstartprojectpwdweb
ThiswillcreatethestructureforaDjangoproject,whichweknowwellbynow.I’llshow
youthefinalstructureoftheinterfaceprojecthere:
$tree-Apwdweb
pwdweb
├──db.sqlite3
├──manage.py
├──pwdweb
│├──__init__.py
│├──settings.py
│├──urls.py
│└──wsgi.py
└──records
├──admin.py
├──forms.py
├──__init__.py
├──migrations
│├──0001_initial.py
│└──__init__.py
├──models.py
├──static
│└──records
│├──css
││└──main.css
│└──js
│├──api.js
│└──jquery-2.1.4.min.js
├──templates
│└──records
│├──base.html
│├──footer.html
│├──home.html
│├──list.html
│├──messages.html
│├──record_add_edit.html
│└──record_confirm_delete.html
├──templatetags
│└──record_extras.py
├──urls.py
└──views.py
Asusual,don’tworryifyoudon’thaveallthefiles,we’lladdthemgradually.Changeto
thepwdwebfolder,andmakesureDjangoiscorrectlysetup:$pythonmanage.py
runserver(ignorethewarningaboutunappliedmigrations).
Shutdowntheserverandcreateanapp:$pythonmanage.pystartapprecords.Thatis
excellent,nowwecanstartcoding.Firstthingsfirst,let’sopenpwdweb/settings.pyand
WOW! eBook
www.wowebook.org
startbyadding'records',attheendoftheINSTALLED_APPtuple(notethatthecommais
includedinthecode).Then,goaheadandfixtheLANGUAGE_CODEandTIME_ZONEsettings
accordingtoyourpreferenceandfinally,addthefollowinglineatthebottom:
ENCRYPTION_KEY=b'qMhPGx-ROWUDr4veh0ybPRL6viIUNe0vcPDmy67x6CQ='
ThisisacustomencryptionkeythathasnothingtodowithDjangosettings,butwewill
needitlateron,andthisisthebestplaceforittobe.Don’tworryfornow,we’llgetback
toit.
WOW! eBook
www.wowebook.org
Themodellayer
Weneedtoaddjustonemodelfortherecordsapplication:Record.Thismodelwill
representeachrecordwewanttostoreinthedatabase:
records/models.py
fromcryptography.fernetimportFernet
fromdjango.confimportsettings
fromdjango.dbimportmodels
classRecord(models.Model):
DEFAULT_ENCODING='utf-8'
title=models.CharField(max_length=64,unique=True)
username=models.CharField(max_length=64)
email=models.EmailField(null=True,blank=True)
url=models.URLField(max_length=255,null=True,blank=True)
password=models.CharField(max_length=2048)
notes=models.TextField(null=True,blank=True)
created=models.DateTimeField(auto_now_add=True)
last_modified=models.DateTimeField(auto_now=True)
defencrypt_password(self):
self.password=self.encrypt(self.password)
defdecrypt_password(self):
self.password=self.decrypt(self.password)
defencrypt(self,plaintext):
returnself.cypher('encrypt',plaintext)
defdecrypt(self,cyphertext):
returnself.cypher('decrypt',cyphertext)
defcypher(self,cypher_func,text):
fernet=Fernet(settings.ENCRYPTION_KEY)
result=getattr(fernet,cypher_func)(
self._to_bytes(text))
returnself._to_str(result)
def_to_str(self,bytes_str):
returnbytes_str.decode(self.DEFAULT_ENCODING)
def_to_bytes(self,s):
returns.encode(self.DEFAULT_ENCODING)
Firstly,wesettheDEFAULT_ENCODINGclassattributeto'utf-8',whichisthemostpopular
typeofencodingfortheweb(andnotonlytheweb).Wesetthisattributeontheclassto
avoidhardcodingastringinmorethanoneplace.
Then,weproceedtosetupallthemodel’sfields.Asyoucansee,Djangoallowsyouto
specifyveryspecificfields,suchasEmailFieldandURLField.Thereasonwhyit’sbetter
tousethesespecificfieldsinsteadofaplainandsimpleCharFieldiswe’llgete-mailand
URLvalidationforfreewhenwecreateaformforthismodel,whichisbrilliant.
WOW! eBook
www.wowebook.org
Alltheoptionsarequitestandard,andwesawtheminChapter10,WebDevelopment
DoneRightbutIwanttopointoutafewthingsanyway.Firstly,titleneedstobeunique
sothateachRecordobjecthasauniquetitleandwedon’twanttoriskhavingdoubles.
Eachdatabasetreatsstringsalittlebitdifferently,accordingtohowitissetup,which
engineitruns,andsoon,soIhaven’tmadethetitlefieldtheprimarykeyforthismodel,
whichwouldhavebeenthenaturalthingtodo.Iprefertoavoidthepainofhavingtodeal
withweirdstringerrorsandIamhappywithlettingDjangoaddaprimarykeytothe
modelautomatically.
Anotheroptionyoushouldunderstandisthenull=True,blank=Truecouple.Theformer
allowsthefieldtobeNULL,whichmakesitnon-mandatory,whilethesecondallowsitto
beblank(thatistosay,anemptystring).TheiruseisquitepeculiarinDjango,soIsuggest
youtotakealookattheofficialdocumentationtounderstandexactlyhowtousethem.
Finally,thedates:createdneedstohaveauto_add_now=True,whichwillsetthecurrent
momentintimeontheobjectwhenit’screated.Ontheotherhand,last_modifiedneeds
tobeupdatedeverytimewesavethemodel,hencewesetauto_now=True.
Afterthefielddefinitions,thereareafewmethodsforencryptinganddecryptingthe
password.Itisalwaysaverybadideatosavepasswordsastheyareinadatabase,
thereforeyoushouldalwaysencryptthembeforesavingthem.
Normally,whensavingapassword,youencryptitusingaonewayencryptionalgorithm
(alsoknownasaonewayhashfunction).Thismeansthat,onceyouhavecreatedthe
hash,thereisnowayforyoutorevertitbacktotheoriginalpassword.
Thiskindofencryptionisnormallyusedforauthentication:theuserputstheirusername
andpasswordinaformand,onsubmission,thecodefetchesthehashfromtheuserrecord
inthedatabaseandcomparesitwiththehashofthepasswordtheuserhasjustputinthe
form.Ifthetwohashesmatch,itmeansthattheywereproducedbythesamepassword,
thereforeauthenticationisgranted.
Inthiscasethough,weneedtobeabletorecoverthepasswords,otherwisethiswhole
applicationwouldn’tbeveryuseful.Therefore,wewilluseaso-calledsymmetric
encryptionalgorithmtoencryptthem.Thewaythisworksisverysimple:thepassword
(calledplaintext)ispassedtoanencryptfunction,alongwithasecretkey.Thealgorithm
producesanencryptedstring(calledcyphertext)outofthem,whichiswhatyoustorein
thedatabase.Whenyouwanttorecoverthepassword,youwillneedthecyphertextand
thesecretkey.Youfeedthemtoadecryptfunction,andyougetbackyouroriginal
password.Thisisexactlywhatweneed.
Inordertoperformsymmetricencryption,weneedthecryptographypackage,whichis
whyIinstructedyoutoinstallit.
AllthemethodsintheRecordclassareverysimple.encrypt_passwordand
decrypt_passwordareshortcutstoencryptanddecryptthepasswordfieldandreassign
theresulttoitself.
Theencryptanddecryptmethodsaredispatchersforthecyphermethod,and_to_str
WOW! eBook
www.wowebook.org
and_to_bytesarejustacoupleofhelpers.Thecryptographylibraryworkswithbytes
objects,soweneedthosehelperstogobackandforthbetweenbytesandstrings,usinga
commonencoding.
Theonlyinterestinglogicisinthecyphermethod.Icouldhavecodeditdirectlyinthe
encryptanddecryptones,butthatwouldhaveresultedinabitofredundancy,andI
wouldn’thavehadthechancetoshowyouadifferentwayofaccessinganobject’s
attribute,solet’sanalyzethebodyofcypher.
WestartbycreatinganinstanceoftheFernetclass,whichprovidesuswiththesymmetric
encryptionfunctionalityweneed.Wesettheinstanceupbypassingthesecretkeyinthe
settings(ENCRYPTION_KEY).Aftercreatingfernet,weneedtouseit.Wecanuseitto
eitherencryptordecrypt,accordingtowhatvalueisgiventothecypher_funcparameter.
Weusegetattrtogetanattributefromanobjectgiventheobjectitselfandthenameof
theattribute.Thistechniqueallowsustofetchanyattributefromanobjectdynamically.
Theresultofgetattr(fernet,cypher_func),withcyper_funcbeing'encrypt',for
example,isthesameasfernet.encrypt.Thegetattrfunctionreturnsamethod,which
wethencallwiththebytesrepresentationofthetextargument.Wethenreturntheresult,
instringformat.
Here’swhatthisfunctionisequivalenttowhenit’scalledbytheencryptdispatcher:
defcypher_encrypt(self,text):
fernet=Fernet(settings.ENCRYPTION_KEY)
result=fernet.encrypt(
self._to_bytes(text))
returnself._to_str(result)
Whenyoutakethetimetounderstanditproperly,you’llseeit’snotasdifficultasit
sounds.
So,wehaveourmodel,henceit’stimetomigrate(Ihopeyourememberthatthiswill
createthetablesinthedatabaseforyourapplication):
$pythonmanage.pymakemigrations
$pythonmanage.pymigrate
Nowyoushouldhaveanicedatabasewithallthetablesyouneedtoruntheinterface
application.Goaheadandcreateasuperuser($pythonmanage.pycreatesuperuser).
Bytheway,ifyouwanttogenerateyourownencryptionkey,itisaseasyasthis:
>>>fromcryptography.fernetimportFernet
>>>Fernet.generate_key()
WOW! eBook
www.wowebook.org
Asimpleform
WeneedaformfortheRecordmodel,sowe’llusetheModelFormtechniquewesawin
Chapter10,WebDevelopmentDoneRight.
records/forms.py
fromdjango.formsimportModelForm,Textarea
from.modelsimportRecord
classRecordForm(ModelForm):
classMeta:
model=Record
fields=['title','username','email','url',
'password','notes']
widgets={'notes':Textarea(
attrs={'cols':40,'rows':4})}
WecreateaRecordFormclassthatinheritsfromModelForm,sothattheformiscreated
automaticallythankstotheintrospectioncapabilitiesofDjango.Weonlyspecifywhich
modeltouse,whichfieldstodisplay(weexcludethedates,whicharehandled
automatically)andweprovideminimalstylingforthedimensionsofthenotesfield,
whichwillbedisplayedusingaTextarea(whichisamultilinetextfieldinHTML).
WOW! eBook
www.wowebook.org
Theviewlayer
Thereareatotaloffivepagesintheinterfaceapplication:home,recordlist,record
creation,recordupdate,andrecorddeleteconfirmation.Hence,therearefiveviewsthat
wehavetowrite.Asyou’llseeinamoment,Djangohelpsusalotbygivingusviewswe
canreusewithminimumcustomization.Allthecodethatfollowsbelongstothe
records/views.pyfile.
Importsandhomeview
Justtobreaktheice,herearetheimportsandtheviewforthehomepage:
fromdjango.contribimportmessages
fromdjango.contrib.messages.viewsimportSuccessMessageMixin
fromdjango.core.urlresolversimportreverse_lazy
fromdjango.views.genericimportTemplateView
fromdjango.views.generic.editimport(
CreateView,UpdateView,DeleteView)
from.formsimportRecordForm
from.modelsimportRecord
classHomeView(TemplateView):
template_name='records/home.html'
WeimportafewtoolsfromDjango.Thereareacoupleofmessaging-relatedobjects,a
URLlazyreverser,andfourdifferenttypesofview.WealsoimportourRecordmodeland
RecordForm.Asyoucansee,theHomeViewclassconsistsofonlytwolinessinceweonly
needtospecifywhichtemplatewewanttouse,therestjustreusesthecodefrom
TemplateView,asitis.It’ssoeasy,italmostfeelslikecheating.
Listingallrecords
Afterthehomeview,wecanwriteaviewtolistalltheRecordinstancesthatwehavein
thedatabase.
classRecordListView(TemplateView):
template_name='records/list.html'
defget(self,request,*args,**kwargs):
context=self.get_context_data(**kwargs)
records=Record.objects.all().order_by('title')#1
forrecordinrecords:
record.plaintext=record.decrypt(record.password)#2
context['records']=records
returnself.render_to_response(context)
Allweneedtodoissub-classTemplateViewagain,andoverridethegetmethod.Weneed
todoacoupleofthings:wefetchalltherecordsfromthedatabaseandsortthembytitle
(#1)andthenparsealltherecordsinordertoaddtheattributeplaintext(#2)ontoeach
ofthem,toshowtheoriginalpasswordonthepage.Anotherwayofdoingthiswouldbe
toaddaread-onlypropertytotheRecordmodel,todothedecryptiononthefly.I’llleave
ittoyou,asafunexercise,toamendthecodetodoit.
WOW! eBook
www.wowebook.org
Afterrecoveringandaugmentingtherecords,weputtheminthecontextdictandfinish
asusualbyinvokingrender_to_response.
Creatingrecords
Here’sthecodeforthecreationview:
classEncryptionMixin:
defform_valid(self,form):
self.encrypt_password(form)
returnsuper(EncryptionMixin,self).form_valid(form)
defencrypt_password(self,form):
self.object=form.save(commit=False)
self.object.encrypt_password()
self.object.save()
classRecordCreateView(
EncryptionMixin,SuccessMessageMixin,CreateView):
template_name='records/record_add_edit.html'
form_class=RecordForm
success_url=reverse_lazy('records:add')
success_message='Recordwascreatedsuccessfully'
Apartofitslogichasbeenfactoredoutinordertobereusedlateronintheupdateview.
Let’sstartwithEncryptionMixin.Allitdoesisoverridetheform_validmethodsothat,
priortosavinganewRecordinstancetothedatabase,wemakesurewecall
encrypt_passwordontheobjectthatresultsfromsavingtheform.Inotherwords,when
theusersubmitstheformtocreateanewRecord,iftheformvalidatessuccessfully,then
theform_validmethodisinvoked.Withinthismethodwhatusuallyhappensisthatan
objectiscreatedoutoftheModelForminstance,likethis:
self.object=form.save()
Weneedtointerferewiththisbehaviorbecauserunningthiscodeasitiswouldsavethe
recordwiththeoriginalpassword,whichisn’tencrypted.Sowechangethistocallsave
ontheformpassingcommit=False,whichcreatestheRecordinstanceoutoftheform,but
doesn’tattempttosaveitinthedatabase.Immediatelyafterwards,weencryptthe
passwordonthatinstanceandthenwecanfinallycallsaveonit,actuallycommittingitto
thedatabase.
Sinceweneedthisbehaviorbothforcreatingandupdatingrecords,Ihavefactoreditout
inamixin.
Note
Perhaps,abettersolutionforthispasswordencryptionlogicistocreateacustomField
(inheritingfromCharFieldistheeasiestwaytodoit)andaddthenecessarylogictoit,so
thatwhenwehandleRecordinstancesfromandtothedatabase,theencryptionand
decryptionlogicisperformedautomaticallyforus.Thoughmoreelegant,thissolution
needsmetodigressandexplainalotmoreaboutDjangointernals,whichistoomuchfor
theextentofthisexample.Asusual,youcantrytodoityourself,ifyoufeellikea
WOW! eBook
www.wowebook.org
challenge.
AftercreatingtheEncryptionMixinclass,wecanuseitintheRecordCreateViewclass.
Wealsoinheritfromtwootherclasses:SuccessMessageMixinandCreateView.The
messagemixinprovidesuswiththelogictoquicklysetupamessagewhencreationis
successful,andtheCreateViewgivesusthenecessarylogictocreateanobjectfroma
form.
Youcanseethatallwehavetocodeissomecustomization:thetemplatename,theform
class,andthesuccessmessageandURL.Everythingelseisgracefullyhandledforusby
Django.
Updatingrecords
ThecodetoupdateaRecordinstanceisonlyatinybitmorecomplicated.Wejustneedto
addsomelogictodecryptthepasswordbeforewepopulatetheformwiththerecorddata.
classRecordUpdateView(
EncryptionMixin,SuccessMessageMixin,UpdateView):
template_name='records/record_add_edit.html'
form_class=RecordForm
model=Record
success_message='Recordwasupdatedsuccessfully'
defget_context_data(self,**kwargs):
kwargs['update']=True
returnsuper(
RecordUpdateView,self).get_context_data(**kwargs)
defform_valid(self,form):
self.success_url=reverse_lazy(
'records:edit',
kwargs={'pk':self.object.pk}
)
returnsuper(RecordUpdateView,self).form_valid(form)
defget_form_kwargs(self):
kwargs=super(RecordUpdateView,self).get_form_kwargs()
kwargs['instance'].decrypt_password()
returnkwargs
Inthisview,westillinheritfrombothEncryptionMixinandSuccessMessageMixin,but
theviewclassweuseisUpdateView.
Thefirstfourlinesarecustomizationasbefore,wesetthetemplatename,theformclass,
theRecordmodel,andthesuccessmessage.Wecannotsetthesuccess_urlasaclass
attributebecausewewanttoredirectasuccessfuledittothesameeditpageforthatrecord
and,inordertodothis,weneedtheIDoftheinstancewe’reediting.Noworries,we’lldo
itanotherway.
First,weoverrideget_context_datainordertoset'update'toTrueinthekwargs
argument,whichmeansthatakey'update'willendupinthecontextdictthatispassed
tothetemplateforrenderingthepage.Wedothisbecausewewanttousethesame
WOW! eBook
www.wowebook.org
templateforcreatingandupdatingarecord,thereforewewillusethisvariableinthe
contexttobeabletounderstandinwhichsituationweare.Thereareotherwaystodothis
butthisoneisquickandeasyandIlikeitbecauseit’sexplicit.
Afteroverridingget_context_data,weneedtotakecareoftheURLredirection.Wedo
thisintheform_validmethodsinceweknowthat,ifwegetthere,itmeanstheRecord
instancehasbeensuccessfullyupdated.Wereversethe'records:edit'view,whichis
exactlytheviewwe’reworkingon,passingtheprimarykeyoftheobjectinquestion.We
takethatinformationfromself.object.pk.
Oneofthereasonsit’shelpfultohavetheobjectsavedontheviewinstanceisthatwecan
useitwhenneededwithouthavingtoalterthesignatureofthemanymethodsintheview
inordertopasstheobjectaround.Thisdesignisveryhelpfulandallowsustoachievea
lotwithveryfewlinesofcode.
Thelastthingweneedtodoistodecryptthepasswordontheinstancebeforepopulating
theformfortheuser.It’ssimpleenoughtodoitintheget_form_kwargsmethod,where
youcanaccesstheRecordinstanceinthekwargsdict,andcalldecrypt_passwordonit.
Thisisallweneedtodotoupdatearecord.Ifyouthinkaboutit,theamountofcodewe
hadtowriteisreallyverylittle,thankstoDjangoclass-basedviews.
Tip
Agoodwayofunderstandingwhichisthebestmethodtooverride,istotakealookatthe
Djangoofficialdocumentationor,evenbetterinthiscase,checkoutthesourcecodeand
lookattheclass-basedviewssection.You’llbeabletoappreciatehowmuchworkhas
beendonetherebyDjangodeveloperssothatyouonlyhavetotouchthesmallestamounts
ofcodetocustomizeyourviews.
Deletingrecords
Ofthethreeactions,deletingarecordisdefinitelytheeasiestone.Allweneedisthe
followingcode:
classRecordDeleteView(SuccessMessageMixin,DeleteView):
model=Record
success_url=reverse_lazy('records:list')
defdelete(self,request,*args,**kwargs):
messages.success(
request,'Recordwasdeletedsuccessfully')
returnsuper(RecordDeleteView,self).delete(
request,*args,**kwargs)
WeonlyneedtoinheritfromSuccessMessageMixinandDeleteView,whichgivesusall
weneed.WesetupthemodelandthesuccessURLasclassattributes,andthenwe
overridethedeletemethodonlytoaddanicemessagethatwillbedisplayedinthelist
view(whichiswhereweredirecttoafterdeletion).
Wedon’tneedtospecifythetemplatename,sincewe’lluseanamethatDjangoinfersby
default:record_confirm_delete.html.
WOW! eBook
www.wowebook.org
Withthisfinalview,we’reallsettohaveaniceinterfacethatwecanusetohandleRecord
instances.
WOW! eBook
www.wowebook.org
SettinguptheURLs
Beforewemoveontothetemplatelayer,let’ssetuptheURLs.Thistime,Iwanttoshow
youtheinclusiontechniqueItalkedaboutinChapter10,WebDevelopmentDoneRight.
pwdweb/urls.py
fromdjango.conf.urlsimportinclude,url
fromdjango.contribimportadmin
fromrecordsimporturlsasrecords_url
fromrecords.viewsimportHomeView
urlpatterns=[
url(r'^admin/',include(admin.site.urls)),
url(r'^records/',include(records_url,namespace='records')),
url(r'^$',HomeView.as_view(),name='home'),
]
ThesearetheURLsforthemainproject.Wehavetheusualadmin,ahomepage,andthen
fortherecordssection,weincludeanotherurls.pyfile,whichwedefineintherecords
application.Thistechniqueallowsforappstobereusableandself-contained.Notethat,
whenincludinganotherurls.pyfile,youcanpassnamespaceinformation,whichyoucan
thenuseinfunctionssuchasreverse,ortheurltemplatetag.Forexample,we’veseen
thatthepathtotheRecordUpdateViewwas'records:edit'.Thefirstpartofthatstringis
thenamespace,andthesecondisthenamethatwehavegiventotheview,asyoucansee
inthefollowingcode:
records/urls.py
fromdjango.conf.urlsimportinclude,url
fromdjango.contribimportadmin
from.viewsimport(RecordCreateView,RecordUpdateView,
RecordDeleteView,RecordListView)
urlpatterns=[
url(r'^add/$',RecordCreateView.as_view(),name='add'),
url(r'^edit/(?P<pk>[0-9]+)/$',RecordUpdateView.as_view(),
name='edit'),
url(r'^delete/(?P<pk>[0-9]+)/$',RecordDeleteView.as_view(),
name='delete'),
url(r'^$',RecordListView.as_view(),name='list'),
]
Wedefinefourdifferenturlinstances.Thereisoneforaddingarecord,whichdoesn’t
needprimarykeyinformationsincetheobjectdoesn’texistyet.Thenwehavetwourl
instancesforupdatinganddeletingarecord,andforthoseweneedtoalsospecifyprimary
keyinformationtobepassedtotheview.SinceRecordinstanceshaveintegerIDs,wecan
safelypassthemontheURL,followinggoodURLdesignpractice.Finally,wedefineone
urlinstanceforthelistofrecords.
Allurlinstanceshavenameinformationwhichisusedinviewsandtemplates.
WOW! eBook
www.wowebook.org
Thetemplatelayer
Let’sstartwiththetemplatewe’lluseasthebasisfortherest:
records/templates/records/base.html
{%loadstaticfromstaticfiles%}
<!DOCTYPEhtml>
<htmllang="en">
<head>
<metacharset="utf-8">
<metaname="viewport"
content="width=device-width,initial-scale=1.0">
<linkhref="{%static"records/css/main.css"%}"
rel="stylesheet">
<title>{%blocktitle%}Title{%endblocktitle%}</title>
</head>
<body>
<divid="page-content">
{%blockpage-content%}{%endblockpage-content%}
</div>
<divid="footer">{%blockfooter%}{%endblockfooter%}</div>
{%blockscripts%}
<script
src="{%static"records/js/jquery-2.1.4.min.js"%}">
</script>
{%endblockscripts%}
</body>
</html>
It’sverysimilartotheoneIusedinChapter10,WebDevelopmentDoneRightalthoughit
isabitmorecompressedandwithonemajordifference.WewillimportjQueryinevery
page.
Note
jQueryisthemostpopularJavaScriptlibraryoutthere.Itallowsyoutowritecodethat
worksonallthemainbrowsersanditgivesyoumanyextratoolssuchastheabilityto
performasynchronouscalls(AJAX)fromthebrowseritself.We’llusethislibraryto
performthecallstotheAPI,bothtogenerateandvalidateourpasswords.Youcan
downloaditathttps://jquery.com/,andputitinthepwdweb/records/static/records/js/
folder(youmayhavetoamendtheimportinthetemplate).
Ihighlightedtheonlyinterestingpartofthistemplateforyou.Notethatweloadthe
JavaScriptlibraryattheend.Thisiscommonpractice,asJavaScriptisusedtomanipulate
thepage,soloadinglibrariesattheendhelpsinavoidingsituationssuchasJavaScript
codefailingbecausetheelementneededhadn’tbeenrenderedonthepageyet.
Homeandfootertemplates
Thehometemplateisverysimple:
records/templates/records/home.html
WOW! eBook
www.wowebook.org
{%extends"records/base.html"%}
{%blocktitle%}WelcometotheRecordswebsite.{%endblock%}
{%blockpage-content%}
<h1>Welcome{{user.first_name}}!</h1>
<divclass="home-option">Tocreatearecordclick
<ahref="{%url"records:add"%}">here.</a>
</div>
<divclass="home-option">Toseeallrecordsclick
<ahref="{%url"records:list"%}">here.</a>
</div>
{%endblockpage-content%}
Thereisnothingnewherewhencomparedtothehome.htmltemplatewesawinChapter
10,WebDevelopmentDoneRight.Thefootertemplateisactuallyexactlythesame:
records/templates/records/footer.html
<divclass="footer">
Goback<ahref="{%url"home"%}">home</a>.
</div>
Listingallrecords
Thistemplatetolistallrecordsisfairlysimple:
records/templates/records/list.html
{%extends"records/base.html"%}
{%loadrecord_extras%}
{%blocktitle%}Records{%endblocktitle%}
{%blockpage-content%}
<h1>Records</h1><spanname="top"></span>
{%include"records/messages.html"%}
{%forrecordinrecords%}
<divclass="record{%cycle'row-light-blue''row-white'%}"
id="record-{{record.pk}}">
<divclass="record-left">
<divclass="record-list">
<spanclass="record-span">Title</span>{{record.title}}
</div>
<divclass="record-list">
<spanclass="record-span">Username</span>
{{record.username}}
</div>
<divclass="record-list">
<spanclass="record-span">Email</span>{{record.email}}
</div>
<divclass="record-list">
<spanclass="record-span">URL</span>
<ahref="{{record.url}}"target="_blank">
{{record.url}}</a>
</div>
<divclass="record-list">
<spanclass="record-span">Password</span>
WOW! eBook
www.wowebook.org
{%hide_passwordrecord.plaintext%}
</div>
</div>
<divclass="record-right">
<divclass="record-list">
<spanclass="record-span">Notes</span>
<textarearows="3"cols="40"class="record-notes"
readonly>{{record.notes}}</textarea>
</div>
<divclass="record-list">
<spanclass="record-span">Lastmodified</span>
{{record.last_modified}}
</div>
<divclass="record-list">
<spanclass="record-span">Created</span>
{{record.created}}
</div>
</div>
<divclass="record-list-actions">
<ahref="{%url"records:edit"pk=record.pk%}">»edit</a>
<ahref="{%url"records:delete"pk=record.pk%}">»delete
</a>
</div>
</div>
{%endfor%}
{%endblockpage-content%}
{%blockfooter%}
<p><ahref="#top">Gobacktotop</a></p>
{%include"records/footer.html"%}
{%endblockfooter%}
Forthistemplateaswell,IhavehighlightedthepartsI’dlikeyoutofocuson.Firstly,I
loadacustomtagsmodule,record_extras,whichwe’llneedlater.Ihavealsoaddedan
anchoratthetop,sothatwe’llbeabletoputalinktoitatthebottomofthepage,toavoid
havingtoscrollallthewayup.
Then,IincludedatemplatetoprovidemewiththeHTMLcodetodisplayDjango
messages.It’saverysimpletemplatewhichI’llshowyoushortly.
Then,wedefinealistofdivelements.EachRecordinstancehasacontainerdiv,inwhich
therearetwoothermaindivelements:record-leftandrecord-right.Inorderto
displaythemsidebyside,Ihavesetthisclassinthemain.cssfile:
.record-left{float:left;width:300px;}
Theoutermostdivcontainer(theonewithclassrecord),hasanidattribute,whichIhave
usedasananchor.Thisallowsustoclickoncancelontherecorddeletepage,sothatif
wechangeourmindsanddon’twanttodeletetherecord,wecangetbacktothelistpage,
andattherightposition.
Eachattributeoftherecordisthendisplayedindivelementswhoseclassisrecord-list.
Mostoftheseclassesarejusttheretoallowmetosetabitofpaddinganddimensionson
WOW! eBook
www.wowebook.org
theHTMLelements.
Thenextinterestingbitisthehide_passwordtag,whichtakestheplaintext,whichisthe
unencryptedpassword.Thepurposeofthiscustomtagistodisplayasequenceof'*'
characters,aslongastheoriginalpassword,sothatifsomeoneispassingbywhileyou’re
onthepage,theywon’tseeyourpasswords.However,hoveringonthatsequenceof'*'
characterswillshowyoutheoriginalpasswordinthetooltip.Here’sthecodeforthe
hide_passwordtag:
records/templatetags/record_extras.py
fromdjangoimporttemplate
fromdjango.utils.htmlimportescape
register=template.Library()
@register.simple_tag
defhide_password(password):
return'<spantitle="{0}">{1}</span>'.format(
escape(password),'*'*len(password))
Thereisnothingfancyhere.Wejustregisterthisfunctionasasimpletagandthenwecan
useitwhereverwewant.Ittakesapasswordandputsitasatooltipofaspanelement,
whosemaincontentisasequenceof'*'characters.Justnoteonething:weneedto
escapethepassword,sothatwe’resureitwon’tbreakourHTML(thinkofwhatmight
happenifthepasswordcontainedadouble-quote`"`,forexample).
Asfarasthelist.htmltemplateisconcerned,thenextinterestingbitisthatwesetthe
readonlyattributetothetextareaelement,soasnottogivetheimpressiontotheuser
thattheycanmodifynotesonthefly.
Then,wesetacoupleoflinksforeachRecordinstance,rightatthebottomofthe
containerdiv.Thereisonefortheeditpage,andanotherforthedeletepage.Notethatwe
needtopasstheurltagnotonlythenamespace:namestring,butalsotheprimarykey
information,asrequiredbytheURLsetupwemadeintheurls.pymoduleforthose
views.
Finally,weimportthefooterandsetthelinktotheanchorontopofthepage.
Now,aspromised,hereisthecodeforthemessages:
records/templates/records/messages.html
{%ifmessages%}
{%formessageinmessages%}
<pclass="{{message.tags}}">{{message}}</p>
{%endfor%}
{%endif%}
Thiscodetakescareofdisplayingmessagesonlywhenthereisatleastonetodisplay.We
givetheptagclassinformationtodisplaysuccessmessagesingreenanderrormessages
inred.
Ifyougrabthemain.cssfilefromthesourcecodeforthebook,youwillnowbeableto
WOW! eBook
www.wowebook.org
visualizethelistpage(yourswillbeblank,youstillneedtoinsertdataintoit),andit
shouldlooksomethinglikethis:
Asyoucansee,Ihavetworecordsinthedatabaseatthemoment.I’mhoveringonthe
passwordofthefirstone,whichismyplatformaccountatmysister’sschool,andthe
passwordisdisplayedinthetooltip.Thedivisionintwodivelements,leftandright,helps
inmakingrowssmallersothattheoverallresultismorepleasingtotheeye.The
importantinformationisontheleftandtheancillaryinformationisontheright.Therow
coloralternatesbetweenaverylightshadeofblueandwhite.
Eachrowhasaneditanddeletelink,atitsbottomleft.We’llshowthepagesforthosetwo
linksrightafterweseethecodeforthetemplatesthatcreatethem.
TheCSScodethatholdsalltheinformationforthisinterfaceisthefollowing:
records/static/records/css/main.css
html,body,*{
font-family:'TrebuchetMS',Helvetica,sans-serif;}
a{color:#333;}
.record{
clear:both;padding:1em;border-bottom:1pxsolid#666;}
.record-left{float:left;width:300px;}
.record-list{padding:2px0;}
.fieldWrapper{padding:5px;}
.footer{margin-top:1em;color:#333;}
.home-option{padding:.6em0;}
WOW! eBook
www.wowebook.org
.record-span{font-weight:bold;padding-right:1em;}
.record-notes{vertical-align:top;}
.record-list-actions{padding:4px0;clear:both;}
.record-list-actionsa{padding:04px;}
#pwd-info{padding:06px;font-size:1.1em;font-weight:bold;}
#id_notes{vertical-align:top;}
/*Messages*/
.success,.errorlist{font-size:1.2em;font-weight:bold;}
.success{color:#25B725;}
.errorlist{color:#B12B2B;}
/*colors*/
.row-light-blue{background-color:#E6F0FA;}
.row-white{background-color:#fff;}
.green{color:#060;}
.orange{color:#FF3300;}
.red{color:#900;}
Pleaseremember,I’mnotaCSSgurusojusttakethisfileasitis,afairlynaivewayto
providestylingtoourinterface.
Creatingandeditingrecords
Nowfortheinterestingpart.Creatingandupdatingarecord.We’llusethesametemplate
forboth,soweexpectsomedecisionallogictobetherethatwilltellusinwhichofthe
twosituationsweare.Asitturnsout,itwillnotbethatmuchcode.Themostexcitingpart
ofthistemplate,however,isitsassociatedJavaScriptfilewhichwe’llexamineright
afterwards.
records/templates/records/record_add_edit.html
{%extends"records/base.html"%}
{%loadstaticfromstaticfiles%}
{%blocktitle%}
{%ifupdate%}Update{%else%}Create{%endif%}Record
{%endblocktitle%}
{%blockpage-content%}
<h1>{%ifupdate%}Updatea{%else%}Createanew{%endif%}
Record
</h1>
{%include"records/messages.html"%}
<formaction="."method="post">{%csrf_token%}
{{form.non_field_errors}}
<divclass="fieldWrapper">{{form.title.errors}}
{{form.title.label_tag}}{{form.title}}</div>
<divclass="fieldWrapper">{{form.username.errors}}
{{form.username.label_tag}}{{form.username}}</div>
<divclass="fieldWrapper">{{form.email.errors}}
{{form.email.label_tag}}{{form.email}}</div>
<divclass="fieldWrapper">{{form.url.errors}}
{{form.url.label_tag}}{{form.url}}</div>
WOW! eBook
www.wowebook.org
<divclass="fieldWrapper">{{form.password.errors}}
{{form.password.label_tag}}{{form.password}}
<spanid="pwd-info"></span></div>
<buttontype="button"id="validate-btn">
ValidatePassword</button>
<buttontype="button"id="generate-btn">
GeneratePassword</button>
<divclass="fieldWrapper">{{form.notes.errors}}
{{form.notes.label_tag}}{{form.notes}}</div>
<inputtype="submit"
value="{%ifupdate%}Update{%else%}Insert{%endif%}">
</form>
{%endblockpage-content%}
{%blockfooter%}
<br>{%include"records/footer.html"%}<br>
Goto<ahref="{%url"records:list"%}">therecordslist</a>.
{%endblockfooter%}
{%blockscripts%}
{{block.super}}
<scriptsrc="{%static"records/js/api.js"%}"></script>
{%endblockscripts%}
Asusual,Ihavehighlightedtheimportantparts,solet’sgothroughthiscodetogether.
Youcanseethefirstbitofdecisionlogicinthetitleblock.Similardecisionlogicisalso
displayedlateron,intheheaderofthepage(theh1HTMLtag),andinthesubmitbutton
attheendoftheform.
Apartfromthislogic,whatI’dlikeyoutofocusonistheformandwhat’sinsideit.Weset
theactionattributetoadot,whichmeansthispage,sothatwedon’tneedtocustomizeit
accordingtowhichviewisservingthepage.Also,weimmediatelytakecareofthecrosssiterequestforgerytoken,asexplainedinChapter10,WebDevelopmentDoneRight.
Notethat,thistime,wecannotleavethewholeformrenderinguptoDjangosincewe
wanttoaddinacoupleofextrathings,sowegodownonelevelofgranularityandask
Djangotorendereachindividualfieldforus,alongwithanyerrors,alongwithitslabel.
Thiswaywestillsavealotofeffort,andatthesametime,wecanalsocustomizetheform
aswelike.Insituationslikethis,it’snotuncommontowriteasmalltemplatetorendera
field,inordertoavoidrepeatingthosethreelinesforeachfield.Inthiscasethough,the
formissosmallIdecidedtoavoidraisingthecomplexitylevelupanyfurther.
Thespanelement,pwd-info,containstheinformationaboutthepasswordthatweget
fromtheAPI.Thetwobuttonsafterthat,validate-btnandgenerate-btn,arehookedup
withtheAJAXcallstotheAPI.
Attheendofthetemplate,inthescriptsblock,weneedtoloadtheapi.jsJavaScript
filewhichcontainsthecodetoworkwiththeAPI.Wealsoneedtouseblock.super,
WOW! eBook
www.wowebook.org
whichwillloadwhatevercodeisinthesameblockintheparenttemplate(forexample,
jQuery).block.superisbasicallythetemplateequivalentofacalltosuper(ClassName,
self)inPython.It’simportanttoloadjQuerybeforeourlibrary,sincethelatterisbased
ontheformer.
TalkingtotheAPI
Let’snowtakealookatthatJavaScript.Idon’texpectyoutounderstandeverything.
Firstly,thisisaPythonbookandsecondly,you’resupposedtobeabeginner(thoughby
now,ninjatrained),sofearnot.However,asJavaScripthas,bynow,becomeessentialif
you’redealingwithawebenvironment,havingaworkingknowledgeofitisextremely
importantevenforaPythondeveloper,sotryandgetthemostoutofwhatI’maboutto
showyou.We’llseethepasswordgenerationfirst:
records/static/records/js/api.js
varbaseURL='http://127.0.0.1:5555/password';
vargetRandomPassword=function(){
varapiURL='{url}/generate'.replace('{url}',baseURL);
$.ajax({
type:'GET',
url:apiURL,
success:function(data,status,request){
$('#id_password').val(data[1]);
},
error:function(){alert('Unexpectederror');}
});
}
$(function(){
$('#generate-btn').click(getRandomPassword);
});
Firstly,wesetavariableforthebaseAPIURL:baseURL.Then,wedefinethe
getRandomPasswordfunction,whichisverysimple.Atthebeginning,itdefinesthe
apiURLextendingbaseURLwithareplacementtechnique.Evenifthesyntaxisdifferent
fromthatofPython,youshouldn’thaveanyissuesunderstandingthisline.
AfterdefiningtheapiURL,theinterestingbitcomesup.Wecall$.ajax,whichisthe
jQueryfunctionthatperformstheAJAXcalls.That$isashortcutforjQuery.Asyoucan
seeinthebodyofthecall,it’saGETrequesttoapiURL.Ifitsucceeds(success:…),an
anonymousfunctionisrun,whichsetsthevalueoftheid_passwordtextfieldtothe
secondelementofthereturneddata.We’llseethestructureofthedatawhenweexamine
theAPIcode,sodon’tworryaboutthatnow.Ifanerroroccurs,wesimplyalerttheuser
thattherewasanunexpectederror.
Note
ThereasonwhythepasswordfieldintheHTMLhasid_passwordastheIDisduetothe
wayDjangorendersforms.Youcancustomizethisbehaviorusingacustomprefix,for
example.Inthiscase,I’mhappywiththeDjangodefaults.
WOW! eBook
www.wowebook.org
Afterthefunctiondefinition,werunacoupleoflinesofcodetobindtheclickeventon
thegenerate-btnbuttontothegetRandomPasswordfunction.Thismeansthat,afterthis
codehasbeenrunbythebrowserengine,everytimeweclickthegenerate-btnbutton,
thegetRandomPasswordfunctioniscalled.
Thatwasn’tsoscary,wasit?Solet’sseewhatweneedforthevalidationpart.
Nowthereisavalueinthepasswordfieldandwewanttovalidateit.Weneedtocallthe
APIandinspectitsresponse.Sincepasswordscanhaveweirdcharacters,Idon’twantto
passthemontheURL,thereforeIwilluseaPOSTrequest,whichallowsmetoputthe
passwordinitsbody.Todothis,Ineedthefollowingcode:
varvalidatePassword=function(){
varapiURL='{url}/validate'.replace('{url}',baseURL);
$.ajax({
type:'POST',
url:apiURL,
data:JSON.stringify({'password':$('#id_password').val()}),
contentType:"text/plain",//AvoidCORSpreflight
success:function(data,status,request){
varvalid=data['valid'],infoClass,grade;
varmsg=(valid?'Valid':'Invalid')+'password.';
if(valid){
varscore=data['score']['total'];
grade=(score<10?'Poor':(score<18?'Medium':'Strong'));
infoClass=(score<10?'red':(score<18?'orange':'green'));
msg+='(Score:{score},{grade})'
.replace('{score}',score).replace('{grade}',grade);
}
$('#pwd-info').html(msg);
$('#pwd-info').removeClass().addClass(infoClass);
},
error:function(data){alert('Unexpectederror');}
});
}
$(function(){
$('#validate-btn').click(validatePassword);
});
Theconceptisthesameasbefore,onlythistimeit’sforthevalidate-btnbutton.The
bodyoftheAJAXcallissimilar.WeuseaPOSTinsteadofaGETrequest,andwedefine
thedataasaJSONobject,whichistheequivalentofusingjson.dumps({'password':
'some_pwd'})inPython.
ThecontentTypelineisaquickhacktoavoidproblemswiththeCORSpreflightbehavior
ofthebrowser.Cross-originresourcesharing(CORS)isamechanismthatallows
restrictedresourcesonawebpagetoberequestedfromanotherdomainoutsideofthe
domainfromwhichtherequestoriginated.Inanutshell,sincetheAPIislocatedat
127.0.0.1:5555andtheinterfaceisrunningat127.0.0.1:8000,withoutthishack,the
browserwouldn’tallowustoperformthecalls.Inaproductionenvironment,youmay
wanttocheckthedocumentationforJSONP,whichisamuchbetter(albeitmore
WOW! eBook
www.wowebook.org
complex)solutiontothisissue.
Thebodyoftheanonymousfunctionwhichisrunifthecallsucceedsisapparentlyonlya
bitcomplicated.Allweneedtodoisunderstandifthepasswordisvalid(from
data['valid']),andassignitagradeandaCSSclassbasedonitsscore.Validityand
scoreinformationcomefromtheAPIresponse.
TheonlytrickybitinthiscodeistheJavaScriptternaryoperator,solet’sseea
comparativeexampleforit:
#Python
error='critical'iferror_level>50else'medium'
//JavaScriptequivalent
error=(error_level>50?'critical':'medium');
Withthisexample,youshouldn’thaveanyissuereadingtherestofthelogicinthe
function.Iknow,Icouldhavejustusedaregularif(...),butJavaScriptcodersusethe
ternaryoperatorallthetime,soyoushouldgetusedtoit.It’sgoodtrainingtoscratchour
headsabitharderinordertounderstandcode.
Lastly,I’dlikeyoutotakealookattheendofthatfunction.Wesetthehtmlofthepwdinfospanelementtothemessageweassembled(msg),andthenwestyleit.Inoneline,
weremovealltheCSSclassesfromthatelement(removeClass()withnoparameters
doesthat),andweaddtheinfoClasstoit.infoClassiseither'red','orange',or
'green'.Ifyougobacktothemain.cssfile,you’llseethematthebottom.
Nowthatwe’veseenboththetemplatecodeandtheJavaScripttomakethecalls,let’ssee
ascreenshotofthepage.We’regoingtoeditthefirstrecord,theoneaboutmysister’s
school.
WOW! eBook
www.wowebook.org
Inthepicture,youcanseethatIupdatedthepasswordbyclickingontheGenerate
Passwordbutton.Then,Isavedtherecord(soyoucouldseethenicemessageontop),
and,finally,IclickedontheValidatePasswordbutton.
Theresultisshowningreenontheright-handsideofthePasswordfield.It’sstrong(23is
actuallythemaximumscorewecanget)sothemessageisdisplayedinaniceshadeof
green.
Deletingrecords
Todeletearecord,gotothelistandclickonthedeletelink.You’llberedirectedtoapage
thatasksyouforconfirmation;youcanthenchoosetoproceedanddeletethepoorrecord,
ortocanceltherequestandgobacktothelistpage.Thetemplatecodeisthefollowing:
records/templates/records/record_confirm_delete.html
{%extends"records/base.html"%}
{%blocktitle%}Deleterecord{%endblocktitle%}
{%blockpage-content%}
<h1>ConfirmRecordDeletion</h1>
<formaction="."method="post">{%csrf_token%}
<p>Areyousureyouwanttodelete"{{object}}"?</p>
<inputtype="submit"value="Confirm"/>&nbsp;
<ahref="{%url"records:list"%}#record-{{object.pk}}">
»cancel</a>
</form>
{%endblockpage-content%}
SincethisisatemplateforastandardDjangoview,weneedtousethenaming
conventionsadoptedbyDjango.Therefore,therecordinquestioniscalledobjectinthe
template.The{{object}}tagdisplaysastringrepresentationfortheobject,whichis
notexactlybeautifulatthemoment,sincethewholelinewillread:Areyousureyou
wanttodelete“Recordobject”?.
Thisisbecausewehaven’taddeda__str__methodtoourModelclassyet,whichmeans
thatPythonhasnoideaofwhattoshowuswhenweaskforastringrepresentationofan
instance.Let’schangethisbycompletingourmodel,addingthe__str__methodatthe
bottomoftheclassbody:
records/models.py
classRecord(models.Model):
...
def__str__(self):
return'{}'.format(self.title)
Restarttheserverandnowthepagewillread:Areyousureyouwanttodelete“Some
Bank”?whereSomeBankisthetitleoftherecordwhosedeletelinkIclickedon.
Wecouldhavejustused{{object.title}},butIprefertofixtherootoftheproblem,
notjusttheeffect.Addinga__str__methodisinfactsomethingthatyououghttodofor
WOW! eBook
www.wowebook.org
allofyourmodels.
Theinterestingbitinthislasttemplateisactuallythelinkforcancelingtheoperation.We
usetheurltagtogobacktothelistview(records:list),butweaddanchorinformation
toitsothatitwilleventuallyreadsomethinglikethis(thisisforpk=2):
http://127.0.0.1:8000/records/#record-2
ThiswillgobacktothelistpageandscrolldowntothecontainerdivthathasIDrecord2,
whichisnice.
Thisconcludestheinterface.Eventhoughthissectionwassimilartowhatwesawin
Chapter10,WebDevelopmentDoneRight,we’vebeenabletoconcentratemoreonthe
codeinthischapter.We’veseenhowusefulDjangoclass-basedviewsare,andweeven
touchedonsomecoolJavaScript.Run$pythonmanage.pyrunserverandyour
interfaceshouldbeupandrunningathttp://127.0.0.1:8000.
Note
Ifyouarewondering,127.0.0.1meansthelocalhost—yourcomputer—while8000is
theporttowhichtheserverisbound,tolistenforincomingrequests.
Nowit’stimetospicethingsupabitwiththesecondpartofthisproject.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
ImplementingtheFalconAPI
ThestructureoftheFalconprojectwe’reabouttocodeisnowherenearasextendedasthe
interfaceone.We’llcodefivefilesaltogether.Inyourch12folder,createanewonecalled
pwdapi.Thisisitsfinalstructure:
$tree-Apwdapi/
pwdapi/
├──core
│├──handlers.py
│└──passwords.py
├──main.py
└──tests
└──test_core
├──test_handlers.py
└──test_passwords.py
TheAPIwasallcodedusingTDD,sowe’realsogoingtoexplorethetests.However,I
thinkit’sgoingtobeeasierforyoutounderstandthetestsifyoufirstseethecode,so
we’regoingtostartwiththat.
WOW! eBook
www.wowebook.org
Themainapplication
ThisisthecodefortheFalconapplication:
main.py
importfalcon
fromcore.handlersimport(
PasswordValidatorHandler,
PasswordGeneratorHandler,
)
validation_handler=PasswordValidatorHandler()
generator_handler=PasswordGeneratorHandler()
app=falcon.API()
app.add_route('/password/validate/',validation_handler)
app.add_route('/password/generate/',generator_handler)
AsintheexampleinChapter10,WebDevelopmentDoneRight,westartbycreatingone
instanceforeachofthehandlersweneed,thenwecreateafalcon.APIobjectand,by
callingitsadd_routemethod,wesetuptheroutingtotheURLsofourAPI.We’llgetto
thedefinitionsofthehandlersinamoment.Firstly,weneedacoupleofhelpers.
WOW! eBook
www.wowebook.org
Writingthehelpers
Inthissection,wewilltakealookatacoupleofclassesthatwe’lluseinourhandlers.It’s
alwaysgoodtofactoroutsomelogicfollowingtheSingleResponsibilityPrinciple.
Note
InOOP,theSingleResponsibilityPrinciple(SRP)statesthateveryclassshouldhave
responsibilityforasinglepartofthefunctionalityprovidedbythesoftware,andthat
responsibilityshouldbeentirelyencapsulatedbytheclass.Allofitsservicesshouldbe
narrowlyalignedwiththatresponsibility.
TheSingleResponsibilityPrincipleistheSinS.O.L.I.D.,anacronymforthefirstfive
OOPandsoftwaredesignprinciplesintroducedbyRobertMartin.
Iheartilysuggestyoutoopenabrowserandreaduponthissubject,itisveryimportant.
Allthecodeinthehelperssectionbelongstothecore/passwords.pymodule.Here’show
itbegins:
frommathimportceil
fromrandomimportsample
fromstringimportascii_lowercase,ascii_uppercase,digits
punctuation='!#$%&()*+-?@_|'
allchars=''.join(
(ascii_lowercase,ascii_uppercase,digits,punctuation))
We’llneedtohandlesomerandomizedcalculationsbutthemostimportantparthereisthe
allowedcharacters.Wewillallowletters,digits,andasetofpunctuationcharacters.To
easewritingthecode,wewillmergethosepartsintotheallcharsstring.
Codingthepasswordvalidator
ThePasswordValidatorclassismyfavoritebitoflogicinthewholeAPI.Itexposesan
is_validandascoremethod.Thelatterrunsalldefinedvalidators(“private”methodsin
thesameclass),andcollectsthescoresintoasingledictwhichisreturnedasaresult.I’ll
writethisclassmethodbymethodsothatitdoesnotgettoocomplicated:
classPasswordValidator:
def__init__(self,password):
self.password=password.strip()
Itbeginsbysettingpassword(withnoleadingortrailingspaces)asaninstanceattribute.
Thiswaywewon’tthenhavetopassitaroundfrommethodtomethod.Allthemethods
thatwillfollowbelongtothisclass.
defis_valid(self):
return(len(self.password)>0and
all(charinallcharsforcharinself.password))
Apasswordisvalidwhenitslengthisgreaterthan0andallofitscharactersbelongtothe
allcharsstring.Whenyoureadtheis_validmethod,it’spracticallyEnglish(that’show
WOW! eBook
www.wowebook.org
amazingPythonis).allisabuilt-infunctionthattellsyouifalltheelementsofthe
iterableyoufeedtoitareTrue.
defscore(self):
result={
'length':self._score_length(),
'case':self._score_case(),
'numbers':self._score_numbers(),
'special':self._score_special(),
'ratio':self._score_ratio(),
}
result['total']=sum(result.values())
returnresult
Thisistheothermainmethod.It’sverysimple,itjustpreparesadictwithalltheresults
fromthevalidators.Theonlyindependentbitoflogichappensattheend,whenwesum
thegradesfromeachvalidatorandassignittoa'total'keyinthedict,justfor
convenience.
Asyoucansee,wescoreapasswordbylength,bylettercase,bythepresenceofnumbers,
andspecialcharacters,and,finally,bytheratiobetweenlettersandnumbers.Lettersallow
acharactertobebetween26*2=52differentpossiblechoices,whiledigitsallowonly
10.Therefore,passwordswhoseletterstodigitsratioishigheraremoredifficulttocrack.
Let’sseethelengthvalidator:
def_score_length(self):
scores_list=([0]*4)+([1]*4)+([3]*4)+([5]*4)
scores=dict(enumerate(scores_list))
returnscores.get(len(self.password),7)
Weassign0pointstopasswordswhoselengthislessthanfourcharacters,1pointfor
thosewhoselengthislessthan8,3foralengthlessthan12,5foralengthlessthan16,
and7foralengthof16ormore.
Inordertoavoidawaterfallofif/elifclauses,Ihaveadoptedafunctionalstylehere.I
preparedascore_list,whichisbasically[0,0,0,0,1,1,1,1,3,...].Then,
byenumeratingit,Igota(length,score)pairforeachlengthlessthan16.Iputthosepairs
intoadict,whichgivesmetheequivalentindictform,soitshouldlooklikethis:{0:0,
1:0,2:0,3:0,4:1,5:1,...}.Ithenperformagetonthisdictwiththelengthofthe
password,settingavalueof7asthedefault(whichwillbereturnedforlengthsof16or
more,whicharenotinthedict).
Ihavenothingagainstif/elifclauses,ofcourse,butIwantedtotaketheopportunityto
showyoudifferentcodingstylesinthisfinalchapter,tohelpyougetusedtoreadingcode
whichdeviatesfromwhatyouwouldnormallyexpect.It’sonlybeneficial.
def_score_case(self):
lower=bool(set(ascii_lowercase)&set(self.password))
upper=bool(set(ascii_uppercase)&set(self.password))
returnint(lowerorupper)+2*(lowerandupper)
Thewaywevalidatethecaseisagainwithanicetrick.lowerisTruewhenthe
WOW! eBook
www.wowebook.org
intersectionbetweenthepasswordandalllowercasecharactersisnon-empty,otherwise
it’sFalse.upperbehavesinthesameway,onlywithuppercasecharacters.
Tounderstandtheevaluationthathappensonthelastline,let’susetheinside-out
techniqueoncemore:lowerorupperisTruewhenatleastoneofthetwoisTrue.When
it’sTrue,itwillbeconvertedtoa1bytheintclass.Thisequatestosaying,ifthereisat
leastonecharacter,regardlessofthecasing,thescoregets1point,otherwiseitstaysat0.
Nowforthesecondpart:lowerandupperisTruewhenbothofthemareTrue,which
meansthatwehaveatleastonelowercaseandoneuppercasecharacter.Thismeansthat,
tocrackthepassword,abrute-forcealgorithmwouldhavetoloopthrough52letters
insteadofjust26.Therefore,whenthat’sTrue,wegetanextratwopoints.
Thisvalidatorthereforeproducesaresultintherange(0,1,3),dependingonwhatthe
passwordis.
def_score_numbers(self):
return2if(set(self.password)&set(digits))else0
Scoringonthenumbersissimpler.Ifwehaveatleastonenumber,wegettwopoints,
otherwiseweget0.Inthiscase,Iusedaternaryoperatortoreturntheresult.
def_score_special(self):
return4if(
set(self.password)&set(punctuation))else0
Thespecialcharactersvalidatorhasthesamelogicasthepreviousonebut,sincespecial
charactersaddquiteabitofcomplexitywhenitcomestocrackingapassword,wehave
scoredfourpointsinsteadofjusttwo.
Thelastonevalidatestheratiobetweenthelettersandthedigits.
def_score_ratio(self):
alpha_count=sum(
1ifc.lower()inascii_lowercaseelse0
forcinself.password)
digits_count=sum(
1ifcindigitselse0forcinself.password)
ifdigits_count==0:
return0
returnmin(ceil(alpha_count/digits_count),7)
Ihighlightedtheconditionallogicintheexpressionsinthesumcalls.Inthefirstcase,we
geta1foreachcharacterwhoselowercaseversionisinascii_lowercase.Thismeans
thatsummingallthose1’supgivesusexactlythecountofalltheletters.Then,wedothe
sameforthedigits,onlyweusethedigitsstringforreference,andwedon’tneedto
lowercasethecharacter.Whendigits_countis0,alpha_count/digits_countwould
causeaZeroDivisionError,thereforewecheckondigits_countandwhenit’s0we
return0.Ifwehavedigits,wecalculatetheceilingoftheletters:digitsratio,andreturnit,
cappedat7.
Ofcourse,therearemanydifferentwaystocalculateascoreforapassword.Myaimhere
isnottogiveyouthefinestalgorithmtodothat,buttoshowyouhowyoucouldgoabout
WOW! eBook
www.wowebook.org
implementingit.
Codingthepasswordgenerator
Thepasswordgeneratorisamuchsimplerclassthanthevalidator.However,Ihavecoded
itsothatwewon’tneedtocreateaninstancetouseit,justtoshowyouyetagaina
differentcodingstyle.
classPasswordGenerator:
@classmethod
defgenerate(cls,length,bestof=10):
candidates=sorted([
cls._generate_candidate(length)
forkinrange(max(1,bestof))
])
returncandidates[-1]
@classmethod
def_generate_candidate(cls,length):
password=cls._generate_password(length)
score=PasswordValidator(password).score()
return(score['total'],password)
@classmethod
def_generate_password(cls,length):
chars=allchars*(ceil(length/len(allchars)))
return''.join(sample(chars,length))
Ofthethreemethods,onlythefirstoneismeanttobeused.Let’sstartouranalysiswith
thelastone:_generate_password.
Thismethodsimplytakesalength,whichisthedesiredlengthforthepasswordwewant,
andcallsthesamplefunctiontogetapopulationoflengthelementsoutofthechars
string.Thereturnvalueofthesamplefunctionisalistoflengthelements,andweneedto
makeitastringusingjoin.
Beforewecancallsample,thinkaboutthis,whatifthedesiredlengthexceedsthelength
ofallchars?ThecallwouldresultinValueError:Samplelargerthanthe
population.
Becauseofthis,wecreatethecharsstringinawaythatitismadebyconcatenatingthe
allcharsstringtoitselfjustenoughtimestocoverthedesiredlength.Togiveyouan
example,let’ssayweneedapasswordof27characters,andlet’spretendallcharsis10
characterslong.length/len(allchars)gives2.7,which,whenpassedtotheceil
function,becomes3.Thismeansthatwe’regoingtoassigncharstoatripleconcatenation
oftheallcharsstring,hencecharswillbe10*3=30characterslong,whichisenough
tocoverourrequirements.
Notethat,inorderforthesemethodstobecalledwithoutcreatinganinstanceofthisclass,
weneedtodecoratethemwiththeclassmethoddecorator.Theconventionisthentocall
thefirstargument,cls,insteadofself,becausePython,behindthescenes,willpassthe
classobjecttothecall.
WOW! eBook
www.wowebook.org
Thecodefor_generate_candidateisalsoverysimple.Wejustgenerateapasswordand,
giventhelength,wecalculateitsscore,andreturnatuple(score,password).
Wedothissothatinthegeneratemethodwecangenerate10(bydefault)passwordseach
timethemethodiscalledandreturntheonethathasthehighestscore.Sinceour
generationlogicisbasedonarandomfunction,it’salwaysagoodwaytoemploya
techniquelikethistoavoidworstcasescenarios.
Thisconcludesthecodeforthehelpers.
WOW! eBook
www.wowebook.org
Writingthehandlers
Asyoumayhavenoticed,thecodeforthehelpersisn’trelatedtoFalconatall.Itisjust
purePythonthatwecanreusewhenweneedit.Ontheotherhand,thecodeforthe
handlersisofcoursebasedonFalcon.Thecodethatfollowsbelongstothe
core/handlers.pymoduleso,aswedidbefore,let’sstartwiththefirstfewlines:
importjson
importfalcon
from.passwordsimportPasswordValidator,PasswordGenerator
classHeaderMixin:
defset_access_control_allow_origin(self,resp):
resp.set_header('Access-Control-Allow-Origin','*')
Thatwasverysimple.Weimportjson,falcon,andourhelpers,andthenwesetupa
mixinwhichwe’llneedinbothhandlers.TheneedforthismixinistoallowtheAPIto
serverequeststhatcomefromsomewhereelse.ThisistheothersideoftheCORScointo
whatwesawintheJavaScriptcodefortheinterface.Inthiscase,weboldlygowhereno
securityexpertwouldeverdare,andallowrequeststocomefromanydomain('*').We
dothisbecausethisisanexerciseand,inthiscontext,itisfine,butdon’tdoitin
production,okay?
Codingthepasswordvalidatorhandler
ThishandlerwillhavetorespondtoaPOSTrequest,thereforeIhavecodedanon_post
method,whichisthewayyoureacttoaPOSTrequestinFalcon.
classPasswordValidatorHandler(HeaderMixin):
defon_post(self,req,resp):
self.process_request(req,resp)
password=req.context.get('_body',{}).get('password')
ifpasswordisNone:
resp.status=falcon.HTTP_BAD_REQUEST
returnNone
result=self.parse_password(password)
resp.body=json.dumps(result)
defparse_password(self,password):
validator=PasswordValidator(password)
return{
'password':password,
'valid':validator.is_valid(),
'score':validator.score(),
}
defprocess_request(self,req,resp):
self.set_access_control_allow_origin(resp)
body=req.stream.read()
ifnotbody:
WOW! eBook
www.wowebook.org
raisefalcon.HTTPBadRequest('Emptyrequestbody',
'AvalidJSONdocumentisrequired.')
try:
req.context['_body']=json.loads(
body.decode('utf-8'))
except(ValueError,UnicodeDecodeError):
raisefalcon.HTTPError(
falcon.HTTP_753,'MalformedJSON',
'JSONincorrectornotutf-8encoded.')
Let’sstartwiththeon_postmethod.Firstofall,wecalltheprocess_requestmethod,
whichdoesasanitycheckontherequestbody.Iwon’tgointofinestdetailbecauseit’s
takenfromtheFalcondocumentation,andit’sastandardwayofprocessingarequest.
Let’sjustsaythat,ifeverythinggoeswell(thehighlightedpart),wegetthebodyofthe
request(alreadydecodedfromJSON)inreq.context['_body'].Ifthingsgobadlyfor
anyreason,wereturnanappropriateerrorresponse.
Let’sgobacktoon_post.Wefetchthepasswordfromtherequestcontext.Atthispoint,
process_requesthassucceeded,butwestilldon’tknowifthebodywasinthecorrect
format.We’reexpectingsomethinglike:{'password':'my_password'}.
Soweproceedwithcaution.Wegetthevalueforthe'_body'keyand,ifthatisnot
present,wereturnanemptydict.Wegetthevaluefor'password'fromthat.Weuseget
insteadofdirectaccesstoavoidKeyErrorissues.
IfthepasswordisNone,wesimplyreturna400error(badrequest).Otherwise,we
validateitandcalculateitsscore,andthensettheresultasthebodyofourresponse.
Youcanseehoweasyitistovalidateandcalculatethescoreofthepasswordinthe
parse_passwordmethod,byusingourhelpers.
Wereturnadictwiththreepiecesofinformation:password,valid,andscore.The
passwordinformationistechnicallyredundantbecausewhoevermadetherequestwould
knowthepasswordbut,inthiscase,Ithinkit’sagoodwayofprovidingenough
informationforthingssuchaslogging,soIaddedit.
WhathappensiftheJSON-decodedbodyisnotadict?Iwillleaveituptoyoutofixthe
code,addingsomelogictocaterforthatedgecase.
Codingthepasswordgeneratorhandler
ThegeneratorhandlerhastohandleaGETrequestwithonequeryparameter:thedesired
passwordlength.
classPasswordGeneratorHandler(HeaderMixin):
defon_get(self,req,resp):
self.process_request(req,resp)
length=req.context.get('_length',16)
resp.body=json.dumps(
PasswordGenerator.generate(length))
defprocess_request(self,req,resp):
self.set_access_control_allow_origin(resp)
WOW! eBook
www.wowebook.org
length=req.get_param('length')
iflengthisNone:
return
try:
length=int(length)
assertlength>0
req.context['_length']=length
except(ValueError,TypeError,AssertionError):
raisefalcon.HTTPBadRequest('Wrongqueryparameter',
'`length`mustbeapositiveinteger.')
Wehaveasimilarprocess_requestmethod.Itdoesasanitycheckontherequest,even
thoughabitdifferentlyfromtheprevioushandler.Thistime,weneedtomakesurethatif
thelengthisprovidedonthequerystring(whichmeans,forexample,http://our-apiurl/?length=23),it’sinthecorrectformat.Thismeansthatlengthneedstobeapositive
integer.
So,tovalidatethat,wedoanintconversion(req.get_param('length')returnsa
string),thenweassertthatlengthisgreaterthanzeroand,finally,weputitincontext
underthe'_length'key.
Doingtheintconversionofastringwhichisnotasuitablerepresentationforaninteger
raisesValueError,whileaconversionfromatypethatisnotastringraisesTypeError,
thereforewecatchthosetwointheexceptclause.
WealsocatchAssertionError,whichisraisedbytheassertlength>0linewhen
lengthisnotapositiveinteger.Wecanthensafelyguaranteethatthelengthisasdesired
withonesingletry/exceptblock.
Tip
Notethat,whencodingatry/exceptblock,youshouldusuallytryandbeasspecificas
possible,separatinginstructionsthatwouldraisedifferentexceptionsifaproblemarose.
Thiswouldallowyoumorecontrolovertheissue,andeasierdebugging.Inthiscase
though,sincethisisasimpleAPI,it’sfinetohavecodewhichonlyreactstoarequestfor
whichlengthisnotintherightformat.
Thecodefortheon_getmethodisquitestraightforward.Itstartsbyprocessingthe
request,thenthelengthisfetched,fallingbackto16(thedefaultvalue)whenit’snot
passed,andthenapasswordisgeneratedanddumpedtoJSON,andthensettobethe
bodyoftheresponse.
WOW! eBook
www.wowebook.org
RunningtheAPI
Inordertorunthisapplication,youneedtorememberthatwesetthebaseURLinthe
interfacetohttp://127.0.0.1:5555.Therefore,weneedthefollowingcommandtostart
theAPI:
$gunicorn-b127.0.0.1:5555main:app
Runningthatwillstarttheappdefinedinthemainmodule,bindingtheserverinstanceto
port5555onlocalhost.FormoreinformationaboutGunicorn,pleaserefertoeither
Chapter10,WebDevelopmentDoneRightordirectlytotheproject’shomepage
(http://gunicorn.org/).
ThecodefortheAPIisnowcompletesoifyouhaveboththeinterfaceandtheAPI
running,youcantrythemouttogether.Seeifeverythingworksasexpected.
WOW! eBook
www.wowebook.org
TestingtheAPI
Inthissection,let’stakealookatthetestsIwroteforthehelpersandforthehandlers.
Testsforthehelpersareheavilybasedonthenose_parameterizedlibrary,asmyfavorite
testingstyleisinterfacetesting,withaslittlepatchingaspossible.Using
nose_parameterizedallowsmetowriteteststhatareeasiertoreadbecausethetestcases
areveryvisible.
Ontheotherhand,testsforthehandlershavetofollowthetestingconventionsforthe
Falconlibrary,sotheywillbeabitdifferent.Thisis,ofcourse,idealsinceitallowsmeto
showyouevenmore.
DuetothelimitedamountofpagesIhaveleft,I’llshowyouonlyapartofthetests,so
makesureyoucheckthemoutinfullinthesourcecode.
Testingthehelpers
Let’sseethetestsforthePasswordGeneratorclass:
tests/test_core/test_passwords.py
classPasswordGeneratorTestCase(TestCase):
deftest__generate_password_length(self):
forlengthinrange(300):
assert_equal(
length,
len(PasswordGenerator._generate_password(length))
)
deftest__generate_password_validity(self):
forlengthinrange(1,300):
password=PasswordGenerator._generate_password(
length)
assert_true(PasswordValidator(password).is_valid())
deftest__generate_candidate(self):
score,password=(
PasswordGenerator._generate_candidate(42))
expected_score=PasswordValidator(password).score()
assert_equal(expected_score['total'],score)
@patch.object(PasswordGenerator,'_generate_candidate')
deftest__generate(self,_generate_candidate_mock):
#checks`generate`returnsthehighestscorecandidate
_generate_candidate_mock.side_effect=[
(16,'&a69Ly+0H4jZ'),
(17,'UXaF4stRfdlh'),
(21,'aB4Ge_KdTgwR'),#thewinner
(12,'IRLT*XEfcglm'),
(16,'$P92-WZ5+DnG'),
(18,'Xi#36jcKA_qQ'),
(19,'?p9avQzRMIK0'),
(17,'4@sY&bQ9*H!+'),
(12,'Cx-QAYXG_Ejq'),
WOW! eBook
www.wowebook.org
(18,'C)RAV(HP7j9n'),
]
assert_equal(
(21,'aB4Ge_KdTgwR'),PasswordGenerator.generate(12))
Withintest__generate_password_lengthwemakesurethe_generate_password
methodhandlesthelengthparametercorrectly.Wegenerateapasswordforeachlengthin
therange[0,300),andverifythatithasthecorrectlength.
Inthetest__generate_password_validitytest,wedosomethingsimilarbut,thistime,
wemakesurethatwhateverlengthweaskfor,thegeneratedpasswordisvalid.Weusethe
PasswordValidatorclasstocheckforvalidity.
Finally,weneedtotestthegeneratemethod.Thepasswordgenerationisrandom,
therefore,inordertotestthisfunction,weneedtomock_generate_candidate,thus
controllingitsoutput.Wesettheside_effectargumentonitsmocktobealistof10
candidates,fromwhichweexpectthegeneratemethodtochoosetheonewiththehighest
score.Settingside_effectonamocktoalistcausesthatmocktoreturntheelementsof
thatlist,oneatatime,eachtimeit’scalled.Toavoidambiguity,thehighestscoreis21,
andonlyonecandidatehasscoredthathigh.Wecallthemethodandmakesurethatthat
particularoneisthecandidatewhichisreturned.
Note
IfyouarewonderingwhyIusedthosedoubleunderscoresinthetestnames,it’svery
simple:thefirstoneisaseparatorandthesecondoneistheleadingunderscorethatispart
ofthenameofthemethodundertest.
TestingthePasswordValidatorclassrequiresmanymorelinesofcode,soI’llshowonly
aportionofthesetests:
pwdapi/tests/test_core/test_passwords.py
fromunittestimportTestCase
fromunittest.mockimportpatch
fromnose_parameterizedimportparameterized,param
fromnose.toolsimport(
assert_equal,assert_dict_equal,assert_true)
fromcore.passwordsimportPasswordValidator,PasswordGenerator
classPasswordValidatorTestCase(TestCase):
@parameterized.expand([
(False,''),
(False,''),
(True,'abcdefghijklmnopqrstuvwxyz'),
(True,'ABCDEFGHIJKLMNOPQRSTUVWXYZ'),
(True,'0123456789'),
(True,'!#$%&()*+-?@_|'),
])
deftest_is_valid(self,valid,password):
validator=PasswordValidator(password)
assert_equal(valid,validator.is_valid())
WOW! eBook
www.wowebook.org
Westartbytestingtheis_validmethod.WetestwhetherornotitreturnsFalsewhenit’s
fedanemptystring,aswellasastringmadeupofonlyspaces,whichmakessurewe’re
testingwhetherwe’recalling.strip()whenweassignthepassword.
Then,weuseallthecharactersthatwewanttobeacceptedtomakesurethefunction
acceptsthem.
Iunderstandthesyntaxbehindtheparameterize.expanddecoratorcanbechallengingat
firstbutreally,allthereistoitisthateachtupleconsistsofanindependenttestcase
which,inturn,meansthatthetest_is_validtestisrunindividuallyforeachtuple,and
thatthetwotupleelementsarepassedtothemethodasarguments:validandpassword.
Wethentestforinvalidcharacters.Weexpectthemalltofailsoweuseparam.explicit,
whichrunsthetestforeachofthecharactersinthatweirdstring.
@parameterized.expand(
param.explicit(char)forcharin'>]{<`\\;,[^/"\'~:}=.'
)
deftest_is_valid_invalid_chars(self,password):
validator=PasswordValidator(password)
assert_equal(False,validator.is_valid())
TheyallevaluatetoFalse,sowe’regood.
@parameterized.expand([
(0,''),#0-3:score0
(0,'a'),#0-3:score0
(0,'aa'),#0-3:score0
(0,'aaa'),#0-3:score0
(1,'aaab'),#4-7:score1
...
(5,'aaabbbbccccddd'),#12-15:score5
(5,'aaabbbbccccdddd'),#12-15:score5
])
deftest__score_length(self,score,password):
validator=PasswordValidator(password)
assert_equal(score,validator._score_length())
Totestthe_score_lengthmethod,Icreated16testcasesforthelengthsfrom0to15.
Thebodyofthetestsimplymakessurethatthescoreisassignedappropriately.
deftest__score_length_sixteen_plus(self):
#allpasswordwhoselengthis16+score7points
password='x'*255
forlengthinrange(16,len(password)):
validator=PasswordValidator(password[:length])
assert_equal(7,validator._score_length())
Theprecedingtestisforlengthsfrom16to254.Weonlyneedtomakesurethatany
lengthafter15gets7asascore.
Iwillskipoverthetestsfortheotherinternalmethodsandjumpdirectlytotheoneforthe
scoremethod.Inordertotestit,Iwanttocontrolexactlywhatisreturnedbyeachofthe
_score_*methodssoImockthemoutandinthetest,Isetareturnvalueforeachofthem.
WOW! eBook
www.wowebook.org
Notethattomockmethodsofaclass,weuseavariantofpatch:patch.object.When
yousetreturnvaluesonmocks,it’snevergoodtohaverepetitionsbecauseyoumaynotbe
surewhichmethodreturnedwhat,andthetestwouldn’tfailinthecaseofaswap.So,
alwaysreturndifferentvalues.Inmycase,Iamusingthefirstfewprimenumberstobe
surethereisnopossibilityofconfusion.
@patch.object(PasswordValidator,'_score_length')
@patch.object(PasswordValidator,'_score_case')
@patch.object(PasswordValidator,'_score_numbers')
@patch.object(PasswordValidator,'_score_special')
@patch.object(PasswordValidator,'_score_ratio')
deftest_score(
self,
_score_ratio_mock,
_score_special_mock,
_score_numbers_mock,
_score_case_mock,
_score_length_mock):
_score_ratio_mock.return_value=2
_score_special_mock.return_value=3
_score_numbers_mock.return_value=5
_score_case_mock.return_value=7
_score_length_mock.return_value=11
expected_result={
'length':11,
'case':7,
'numbers':5,
'special':3,
'ratio':2,
'total':28,
}
validator=PasswordValidator('')
assert_dict_equal(expected_result,validator.score())
Iwanttopointoutexplicitlythatthe_score_*methodsaremocked,soIsetupmy
validatorinstancebypassinganemptystringtotheclassconstructor.Thismakesiteven
moreevidenttothereaderthattheinternalsoftheclasshavebeenmockedout.Then,I
justcheckiftheresultisthesameaswhatIwasexpecting.
ThislasttestistheonlyoneinthisclassinwhichIusedmocks.Alltheothertestsforthe
_score_*methodsareinaninterfacestyle,whichreadsbetterandusuallyproducesbetter
results.
Testingthehandlers
Let’sbrieflyseeoneexampleofatestforahandler:
pwdapi/tests/test_core/test_handlers.py
importjson
fromunittest.mockimportpatch
fromnose.toolsimportassert_dict_equal,assert_equal
WOW! eBook
www.wowebook.org
importfalcon
importfalcon.testingastesting
fromcore.handlersimport(
PasswordValidatorHandler,PasswordGeneratorHandler)
classPGHTest(PasswordGeneratorHandler):
defprocess_request(self,req,resp):
self.req,self.resp=req,resp
returnsuper(PGHTest,self).process_request(req,resp)
classPVHTest(PasswordValidatorHandler):
defprocess_request(self,req,resp):
self.req,self.resp=req,resp
returnsuper(PVHTest,self).process_request(req,resp)
BecauseofthetoolsFalcongivesyoutotestyourhandlers,Icreatedachildforeachof
theclassesIwantedtotest.TheonlythingIchanged(byoverridingamethod)isthatin
theprocess_requestmethod,whichiscalledbybothclasses,beforeprocessingthe
requestImakesureIsetthereqandrespargumentsontheinstance.Thenormalbehavior
oftheprocess_requestmethodisthusnotalteredinanyotherway.Bydoingthis,
whateverhappensoverthecourseofthetest,I’llbeabletocheckagainstthoseobjects.
It’squitecommontousetrickslikethiswhentesting.Weneverchangethecodetoadapt
foratest,itwouldbebadpractice.Wefindawayofadaptingourteststosuitourneeds.
classTestPasswordValidatorHandler(testing.TestBase):
defbefore(self):
self.resource=PVHTest()
self.api.add_route('/password/validate/',self.resource)
ThebeforemethodiscalledbytheFalconTestBaselogic,anditallowsustosetupthe
resourcewewanttotest(thehandler)andarouteforit(whichisnotnecessarilythesame
astheoneweuseinproduction).
deftest_post(self):
self.simulate_request(
'/password/validate/',
body=json.dumps({'password':'abcABC0123#&'}),
method='POST')
resp=self.resource.resp
assert_equal('200OK',resp.status)
assert_dict_equal(
{'password':'abcABC0123#&',
'score':{'case':3,'length':5,'numbers':2,
'special':4,'ratio':2,'total':16},
'valid':True},
json.loads(resp.body))
Thisisthetestforthehappypath.AllitdoesissimulateaPOSTrequestwithaJSON
payloadasbody.Then,weinspecttheresponseobject.Inparticular,weinspectitsstatus
anditsbody.Wemakesurethatthehandlerhascorrectlycalledthevalidatorandreturned
itsresults.
WOW! eBook
www.wowebook.org
Wealsotestthegeneratorhandler:
classTestPasswordGeneratorHandler(testing.TestBase):
defbefore(self):
self.resource=PGHTest()
self.api.add_route('/password/generate/',self.resource)
@patch('core.handlers.PasswordGenerator')
deftest_get(self,PasswordGenerator):
PasswordGenerator.generate.return_value=(7,'abc123')
self.simulate_request(
'/password/generate/',
query_string='length=7',
method='GET')
resp=self.resource.resp
assert_equal('200OK',resp.status)
assert_equal([7,'abc123'],json.loads(resp.body))
Forthisoneaswell,Iwillonlyshowyouthetestforthehappypath.Wemockoutthe
PasswordGeneratorclassbecauseweneedtocontrolwhichpassworditwillgenerateand,
unlesswemock,wewon’tbeabletodoit,asitisarandomprocess.
Oncewehavecorrectlysetupitsreturnvalue,wecansimulatetherequestagain.Inthis
case,it’saGETrequest,withadesiredlengthof7.Weuseatechniquesimilartotheone
weusedfortheotherhandler,andchecktheresponsestatusandbody.
ThesearenottheonlytestsyoucouldwriteagainsttheAPI,andthestylecouldbe
differentaswell.Somepeoplemockoften,ItendtomockonlywhenIreallyhaveto.Just
trytoseeifyoucanmakesomesenseoutofthem.Iknowthey’renotreallyeasybut
they’llbegoodtrainingforyou.Testsareextremelyimportantsogiveityourbestshot.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Wheredoyougofromhere?
Ifyoulikedthisprojectandyoufeellikeexpandingit,hereareafewsuggestions:
ImplementtheencryptioninthemechanismofacustomDjangofield.
Amendthetemplatefortherecordlistsothatyoucansearchforaparticularrecord.
AmendtheJavaScripttouseJSONPwithacallbacktoovercometheCORSissue.
AmendtheJavaScripttofirethevalidationcallwhenthepasswordfieldchanges.
WriteaDjangocommandthatallowsyoutoencryptanddecryptthedatabasefile.
Whenyoudoitfromthecommandline,incorporatethatbehaviorintothewebsite,
possiblyonthehomepage,sothatyoudon’thaveaccesstotherecordsunlessyou
areauthenticated.Thisisdefinitelyahardchallengeasitrequireseitheranother
databasewithanauthenticationpasswordstoredproperlywithaonewayhash,or
someseriousreworkingofthedatastructureusedtoholdtherecordmodeldata.
Evenifyoudon’thavethemeanstodoitnow,tryandthinkabouthowyouwould
solvethisproblem.
SetupPostgreSQLonyourmachineandswitchtousingitinsteadoftheSQLitefile
thatisthedefault.
Addtheabilitytoattachafiletoarecord.
Playwiththeapplication,trytofindoutwhichfeaturesyouwanttoaddorchange,
andthendoit.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Summary
Inthischapter,we’veworkedonafinalprojectthatinvolvesaninterfaceandanAPI.We
haveusedtwodifferentframeworkstoaccomplishourgoal:DjangoandFalcon.Theyare
verydifferentandhaveallowedustoexploredifferentconceptsandtechniquestocraft
oursoftwareandmakethisfunapplicationcomealive.
Wehaveseenanexampleofsymmetricencryptionandexploredcodethatwaswrittenina
morefunctionalstyle,asopposedtoamoreclassiccontrolflow-orientedapproach.We
havereusedandextendedtheDjangoclass-basedviews,reducingtoaminimumthe
amountofcodewehadtowrite.
WhencodingtheAPI,wedecoupledhandlingrequestsfrompasswordmanagement.This
wayit’smucheasiertoseewhichpartofthecodedependsontheFalconframeworkand
whichisindependentfromit.
Finally,wesawafewtestsforthehelpersandhandlersoftheAPI.Wehavebriefly
touchedonatechniquethatIusetoexpandclassesundertestinordertobeabletotest
againstthosepartsofthecodewhichwouldnotnormallybeavailable.
Myaiminthischapterwastoprovideyouwithaninterestingexamplethatcouldbe
expandedandimprovedindifferentways.Ialsowantedtogiveyouafewexamplesof
differentcodingstylesandtechniques,whichiswhyIchosetospreadthingsapartanduse
differentframeworks.
WOW! eBook
www.wowebook.org
WOW! eBook
www.wowebook.org
Awordoffarewell
Ihopethatyouarestillthirstyandthatthisbookwillbejustthefirstofmanystepsyou
taketowardsPython.It’satrulywonderfullanguage,wellworthlearningdeeply.
Ihopethatyouenjoyedthisjourneywithme,Ididmybesttomakeitinterestingforyou.
Itsurewasforme,Ihadsuchagreattimewritingthesepages.
Pythonisopensource,sopleasekeepsharingitandconsidersupportingthewonderful
communityaroundit.
Tillnexttime,myfriend,farewell!
WOW! eBook
www.wowebook.org
Index
A
adhocpolymorphism
about/Polymorphism–abriefoverview
AJAX
about/Thefutureofwebdevelopment,Thetemplatelayer
Anaconda
URL/Wheredowegofromhere?
anonymousfunctions
about/Anonymousfunctions
API
testing/TestingtheAPI
helpers,testing/Testingthehelpers
handlers,testing/Testingthehandlers
application
testing/Testingyourapplication
applicationprogramminginterface(API)
about/Writingaunittest
assertion
about/Assertions
WOW! eBook
www.wowebook.org
B
Base64
about/Theimports
baseclass
about/Inheritanceandcomposition
binarysearch
about/Wheretoinspect
Bitbucket
URL/Exceptions
black-boxtests
about/Testingyourapplication
Bokeh
URL/Wheredowegofromhere?
boundaries
about/Aspecializedelse:elif
boundary
about/Boundariesandgranularity
built-inexceptionshierarchy
URL/Exceptions
built-infunctions
about/Built-infunctions
built-inscope
about/Scopes
businesslogic,GUIapplication
webpage,fetching/Thebusinesslogic,Fetchingthewebpage
images,saving/Savingtheimages
user,alerting/Alertingtheuser
WOW! eBook
www.wowebook.org
C
callback
about/Thelayoutlogic
campaign
about/Preparingthedata
classes
about/Objectandclasses,Object-orientedprogramming
classmethods
about/Classmethods
code
writing,guidelines/Guidelinesonhowtowritegoodcode
documenting/Documentingyourcode
collectionsmodule
about/Thecollectionsmodule
namedtuples/Namedtuples
defaultdict/Defaultdict
ChainMap/ChainMap
comma-separatedvalues(CSV)
about/SavingtheDataFrametoafile
commandprompt
about/SettingupthePythoninterpreter
composition
about/Inheritanceandcomposition
comprehensions
about/map,zip,andfilter,Comprehensions
nestedcomprehensions/Nestedcomprehensions
filtering/Filteringacomprehension
dictcomprehensions/dictcomprehensions
setcomprehensions/setcomprehensions
andgenerators/Don’toverdocomprehensionsandgenerators
conditionalprogramming
about/Conditionalprogramming
elif/Aspecializedelse:elif
ternaryoperator/Theternaryoperator
considerations
threading/Threadingconsiderations
console
about/SettingupthePythoninterpreter,Yourfriend,theconsole
constructor
about/Initializinganinstance
containerdatatypes
defining/Thecollectionsmodule
contextmanager
WOW! eBook
www.wowebook.org
using/Thebusinesslogic
contextobject
preparing/Thetemplatelayer
cookies
about/HowdoestheWebwork?
CPC(CostPerClick)
about/Unpackingtheuserdata
CPI(CostPerImpression)
about/Unpackingtheuserdata
Cron
about/RunningPythonscripts
Cross-originresourcesharing(CORS)
about/TalkingtotheAPI
cross-siterequestforgery(CSRF)attack
about/Creatingtheform
CSS(CascadingStyleSheets)
about/Aregexwebsite
CTR(ClickThroughRate)
about/Unpackingtheuserdata
customexceptions
writing/Exceptions
customiterator
writing/Writingacustomiterator
cypertext
about/Themodellayer
WOW! eBook
www.wowebook.org
D
data
dealingwith/Dealingwithdata
notebook,settingup/Settingupthenotebook
preparing/Preparingthedata
cleaning/Cleaningthedata
DataFrame,creating/CreatingtheDataFrame
DataFrame,savingtofile/SavingtheDataFrametoafile
results,visualizing/Visualizingtheresults
DataFrame
campaignname,unpacking/Unpackingthecampaignname
userdata,unpacking/Unpackingtheuserdata
defining/Cleaningeverythingup
datamigrations
about/AddingtheEntrymodel
datastructures
selecting/Howtochoosedatastructures
debuggingtechniques
about/Debuggingtechniques
debugging,withprint/Debuggingwithprint
debugging,withcustomfunction/Debuggingwithacustomfunction
traceback,inspecting/Inspectingthetraceback
Pythondebugger,using/UsingthePythondebugger
logfiles,inspecting/Inspectinglogfiles
profiling/Profiling
assertions/Assertions
information,finding/Wheretofindinformation
decorate-sort-undecorate
about/map
URL/map
decoration
about/Decorators
decorationpoint
about/Decorators
decorator
about/Decorators
decoratorfactory
about/Adecoratorfactory
decorators
about/Decorators
defaultvalues
about/Keywordargumentsanddefaultvalues
deterministprofiling
WOW! eBook
www.wowebook.org
about/ProfilingPython
diamond
about/Multipleinheritance
dictionaries
about/Mappingtypes–dictionaries
discounts
applying/Example2–applyingdiscounts
dispatcher
about/Example2–applyingdiscounts
Django
settingup/SettingupDjango
URL/SettingupDjango
project,starting/Startingtheproject
users,creating/Creatingusers
about/Writingthetemplates
Djangointerface
implementing/Ourimplementation,ImplementingtheDjangointerface
setup/Thesetup
modellayer/Themodellayer
simpleform/Asimpleform
viewlayer/Theviewlayer
URLs,settingup/SettinguptheURLs
templatelayer/Thetemplatelayer
DjangoURLdispatcher
defining/TheDjangoURLdispatcher
regularexpression/Regularexpressions
Djangowebframework
defining/TheDjangowebframework
modellayer/Themodellayer
viewlayer/Theviewlayer
templatelayer/Thetemplatelayer
docstrings
about/Documentingyourcode
DRY(Don’tRepeatYourself)principle/Howdoweusemodulesandpackages
dunder
about/Goingbeyondnext
WOW! eBook
www.wowebook.org
E
enclosingscope
about/Scopes
entries
about/Startingtheproject
environment
settingup/Settinguptheenvironment
equalities
about/Aspecializedelse:elif
Euclid’salgorithm
about/Don’toverdocomprehensionsandgenerators
Euclideanformula
about/Don’toverdocomprehensionsandgenerators
exception
about/Aspecialelseclause
exceptions
about/Exceptions
executionmodel,Python
about/Python’sexecutionmodel
WOW! eBook
www.wowebook.org
F
factorial
about/Howdoweusemodulesandpackages
Falcon
JSONquoteserver,building/BuildingaJSONquoteserverinFalcon
about/BuildingaJSONquoteserverinFalcon
URL/BuildingaJSONquoteserverinFalcon
FalconAPI
implementing/ImplementingtheFalconAPI
mainapplication/Themainapplication
helpers,writing/Writingthehelpers
handlers,writing/Writingthehandlers
API,running/RunningtheAPI
API,testing/TestingtheAPI
features,Python
portability/Portability
coherence/Coherence
developerproductivity/Developerproductivity
extensivelibrary/Anextensivelibrary
softwarequality/Softwarequality
softwareintegration/Softwareintegration
satisfactionandenjoyment/Satisfactionandenjoyment
Fibonaccisequenceexample
about/Onelastexample
filter
about/map,zip,andfilter
defining/filter
Flask
URL/WritingaFlaskview
about/WritingaFlaskview
Flaskview
writing/WritingaFlaskview
floatingpointnumbers
URL/Reals
form
about/Creatingtheform
framework
about/TheDjangowebframework
function
about/Howdoweusemodulesandpackages
functionattributes
about/Functionattributes
functions
WOW! eBook
www.wowebook.org
using/Whyusefunctions?
codeduplication,reducing/Reducecodeduplication
complextask,splitting/Splittingacomplextask
implementationdetails,hiding/Hideimplementationdetails
readability,improving/Improvereadability
traceability,improving/Improvetraceability
writing,tips/Afewusefultips
example/Onefinalexample
WOW! eBook
www.wowebook.org
G
generationbehavior,inbuilt-ins
about/Generationbehaviorinbuilt-ins
generatorexpressions
about/Generatorexpressions
generatorfunctions
about/Generatorfunctions
generatorobjects
about/Goingbeyondnext
generators
about/map,zip,andfilter,Generators
generatorfunctions/Generators
generatorexpressions/Generators
andcomprehensions/Don’toverdocomprehensionsandgenerators
getter
about/Thepropertydecorator
gettersandsetters
about/Thepropertydecorator
Git
about/Guidelinesonhowtowritegoodcode
GitHub
URL/Exceptions
globalscope
about/Scopes
gray-boxtesting
about/Testingyourapplication
greatestcommondivisor(GCD)
about/Don’toverdocomprehensionsandgenerators
Greenphase
about/Test-drivendevelopment
GUI(GraphicalUserInterface)/RunningPythonasaGUIapplication
GUIapplication
Python,runningas/RunningPythonasaGUIapplication
about/Secondapproach–aGUIapplication
imports/Secondapproach–aGUIapplication,Theimports
layoutlogic/Thelayoutlogic
businesslogic/Thebusinesslogic
improving/Howtoimprovetheapplication?
Gunicorn
URL/RunningtheAPI
Gunicorn(GreenUnicorn)
about/BuildingaJSONquoteserverinFalcon
WOW! eBook
www.wowebook.org
H
handlers
writing/Writingthehandlers
passwordvalidatorhandler,coding/Codingthepasswordvalidatorhandler
passwordgeneratorhandler,coding/Codingthepasswordgeneratorhandler
Hashability
about/Settypes
helpers
writing/Writingthehelpers
passwordvalidator,coding/Codingthepasswordvalidator
passwordgenerator,coding/Codingthepasswordgenerator
HTMLDocumentObjectModel(DOM)
about/Creatingtheform
HypertextMarkupLanguage(HTML)
about/Theviewlayer
HypertextTransferProtocol(HTTP)
about/WhatistheWeb?
WOW! eBook
www.wowebook.org
I
IDLE(IntegratedDeveLopmentEnvironment)/RunningthePythoninteractiveshell
immutable
about/Aproperintroduction,Mutableorimmutable?Thatisthequestion
immutablesequences
about/Immutablesequences
stringsandbytes/Stringsandbytes
tuples/Tuples
implicitconcatenation
about/BuildingaJSONquoteserverinFalcon
in-place
about/Unpackingtheuserdata
indexing
about/Aboutindexingandslicing
inequalities
about/Aspecializedelse:elif
infiniteloop
about/Thewhileloop
inheritance
about/Inheritanceandcomposition
initializer
about/Objectandclasses,Initializinganinstance
inner
about/Aspecializedelse:elif
inputparameters
about/Inputparameters
argumentpassing/Argumentpassing
assignment,toargumentnames/Assignmenttoargumentnamesdon’taffectthe
caller
mutable,changing/Changingamutableaffectsthecaller
specifying/Howtospecifyinputparameters
positionalarguments/Positionalarguments
keywordarguments/Keywordargumentsanddefaultvalues
defaultvalues/Keywordargumentsanddefaultvalues
variablepositionalarguments/Variablepositionalarguments
variablekeywordarguments/Variablekeywordarguments
keyword-onlyarguments/Keyword-onlyarguments
,combining/Combininginputparameters
mutabledefaults/Avoidthetrap!Mutabledefaults
inside-outtechnique
about/Iteratingoverasequence
installing
Python/InstallingPython
WOW! eBook
www.wowebook.org
instanceattributes
about/Classandobjectnamespaces
instancesofclasses
about/Objectandclasses
integerdivision(//)
about/Integers
IntegratedDevelopmentEnvironments(IDEs)/AnoteontheIDEs
about/Thepropertydecorator
interfacetesting
about/Interfacetesting
Internet
about/WhatistheWeb?
ipdblibrary
about/UsingthePythondebugger
IPython
URL/Wheredowegofromhere?
Ipython
about/IPythonandJupyternotebook
URL/IPythonandJupyternotebook
iterable
about/Writingacustomiterator
iterator
about/Iteratorsanditerables,Writingacustomiterator
itertoolsmodule
about/Aquickpeekattheitertoolsmodule
infiniteiterators/Infiniteiterators
terminating,onshortestinputsequence/Iteratorsterminatingontheshortest
inputsequence
combinatoricgenerators/Combinatoricgenerators
WOW! eBook
www.wowebook.org
J
jQuery
about/Thetemplatelayer
URL/Thetemplatelayer
JSON(JavaScriptObjectNotation)
about/Exceptions
JSONquoteserver
building,inFalcon/BuildingaJSONquoteserverinFalcon
Jupyternotebook
about/IPythonandJupyternotebook
URL/IPythonandJupyternotebook
WOW! eBook
www.wowebook.org
K
KeepassX
about/Thechallenge
keyword-onlyparameter
about/Keyword-onlyarguments
keywordarguments
about/Keywordargumentsanddefaultvalues
WOW! eBook
www.wowebook.org
L
lambdas
about/Anonymousfunctions
library
about/Howdoweusemodulesandpackages
listcomprehension
about/Lists
local,enclosing,global,built-in(LEGB)/Scopes
localscope
about/Scopes
logfiles
about/Inspectinglogfiles
loggers/Inspectinglogfiles
handlers/Inspectinglogfiles
filters/Inspectinglogfiles
formatters/Inspectinglogfiles
logging
URL/Inspectinglogfiles
loop
iterating,overrange/Iteratingoverarange
iterating,oversequence/Iteratingoverasequence
looping
about/Looping
forloop/Theforloop
iteratorsanditerables/Iteratorsanditerables
iterating,overmultiplesequences/Iteratingovermultiplesequences
whileloop/Thewhileloop
breakandcontinuestatements/Thebreakandcontinuestatements
elseclause/Aspecialelseclause
WOW! eBook
www.wowebook.org
M
magicmethod
about/Objectandclasses
magicmethods
about/Goingbeyondnext
map
about/map,zip,andfilter
defining/map
markdown
about/IPythonandJupyternotebook
masterpassword
about/Thechallenge
Matplotlib
URL/Wheredowegofromhere?
Mercurial
about/Guidelinesonhowtowritegoodcode
mergeandinsertionsort
about/Lists
metaclasses
about/Objectandclasses,ThesimplestPythonclass
metaprogramming
about/ThesimplestPythonclass
method
about/Goingbeyondnext
methodresolutionorder(MRO)
about/Methodresolutionorder
methods
about/Aproperintroduction
middlewareclass
about/Writingthetemplates
migration
about/AddingtheEntrymodel
migrations
applying/Startingtheproject
mixins
about/Multipleinheritance
mocks
about/Mockobjectsandpatching
model
about/Themodellayer
model-template-view(MTV)pattern
about/Djangodesignphilosophy
model-view-controller(MVC)
WOW! eBook
www.wowebook.org
about/Djangodesignphilosophy
modules
using/Howdoweusemodulesandpackages
mutable
about/Aproperintroduction,Mutableorimmutable?Thatisthequestion
mutablesequences
about/Mutablesequences
lists/Lists
bytearrays/Bytearrays
WOW! eBook
www.wowebook.org
N
NameErrorexception/Scopes
namelocalization
about/Namelocalization
namemangling
about/Privatemethodsandnamemangling
names
about/Namesandnamespaces
defining/Aboutthenames
namespaces
about/Namesandnamespaces
nano
about/Usingconsoleeditors
negativeindexing
about/Aboutindexingandslicing
nose-parameterized
URL/Amoreinterestingexample
Numba
URL/Wheredowegofromhere?
numbers
about/Numbers
integers/Integers
booleans/Booleans
realnumbers/Reals
complexnumbers/Complexnumbers
fractionsanddecimals/Fractionsanddecimals
NumericPython
about/CreatingtheDataFrame
NumPy
URL/Wheredowegofromhere?
WOW! eBook
www.wowebook.org
O
object
defining/Everythingisanobject
mutable/Mutableorimmutable?Thatisthequestion
immutable/Mutableorimmutable?Thatisthequestion
object-orientedprogramming
about/Object-orientedprogramming
Pythonclass/ThesimplestPythonclass
classandobjectnamespaces/Classandobjectnamespaces
attributeshadowing/Attributeshadowing
selfvariable,using/I,me,andmyself–usingtheselfvariable
instance,initializing/Initializinganinstance
code,reusing/OOPisaboutcodereuse
inheritanceandcomposition/Inheritanceandcomposition
baseclass,accessing/Accessingabaseclass
multipleinheritance/Multipleinheritance
methodresolutionorder/Methodresolutionorder
classandstaticmethods/Staticandclassmethods
privatemethods/Privatemethodsandnamemangling
namemangling/Privatemethodsandnamemangling
propertydecorator/Thepropertydecorator
operatoroverloading/Operatoroverloading
polymorphism/Polymorphism–abriefoverview
object-relationalmapping(ORM)
about/Themodellayer
objects
about/Aproperintroduction,Objectandclasses,Object-orientedprogramming
importing/Importingobjects
relativeimports/Relativeimports
onewayencryptionalgorithm
about/Themodellayer
onewayhashfunction
about/Themodellayer
operatingsystem(OS)
about/Portability
operatoroverloading
about/Lists,Operatoroverloading
outer
about/Aspecializedelse:elif
WOW! eBook
www.wowebook.org
P
package
about/HowisPythoncodeorganized
packages
using/Howdoweusemodulesandpackages
Pandas
URL/Wheredowegofromhere?
patching
about/Mockobjectsandpatching
URL/Aclassicunittestexample
pdb
about/UsingthePythondebugger
PEP328
about/Relativeimports
performanceconsiderations
about/Someperformanceconsiderations
pivottable
about/Visualizingtheresults
plaintext
about/Themodellayer
polymorphism/Polymorphism–abriefoverview
primarykey
about/Themodellayer,AddingtheEntrymodel
primegenerator
about/Example1–aprimegenerator
primitive
about/Don’toverdocomprehensionsandgenerators
principleofleastastonishment
about/Theprincipleofleastastonishment
principles,Djangowebframework
DRY/Djangodesignphilosophy
Loosecoupling/Djangodesignphilosophy
Lesscode/Djangodesignphilosophy
Consistency/Djangodesignphilosophy
profiling
about/Exceptions,Whentoprofile?
properties
about/Aproperintroduction
protocols
about/WhatistheWeb?
pullprotocol
about/HowdoestheWebwork?
pushprotocol
WOW! eBook
www.wowebook.org
about/HowdoestheWebwork?
PyGTK
about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTK
PyPy
URL/Whatarethedrawbacks?
PyQt
about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTK
pytest
URL/Amoreinterestingexample
Pythagoreantriple
about/Filteringacomprehension
Python
about/EnterthePython
features/AboutPython,Softwareintegration
drawbacks/Whatarethedrawbacks?
users/WhoisusingPythontoday?
installing/InstallingPython
references/SettingupthePythoninterpreter
running,asservice/RunningPythonasaservice
running,asGUIapplication/RunningPythonasaGUIapplication
executionmodel/Python’sexecutionmodel
profiling/ProfilingPython
Python2
versusPython3/Python2versusPython3–thegreatdebate
Pythoncode
organizing/HowisPythoncodeorganized
Pythonculture
about/ThePythonculture
PythonEnhancementProposal(PEP)/Guidelinesonhowtowritegoodcode
Pythoninteractiveshell
running/RunningthePythoninteractiveshell
Pythoninterpreter
settingup/SettingupthePythoninterpreter
Pythonmodule
reference/Everythingisanobject
PythonPackageIndex(PyPI)/Anextensivelibrary
Pythonprogram
running/HowyoucanrunaPythonprogram
Pythonscripts
running/RunningPythonscripts
WOW! eBook
www.wowebook.org
Q
qualityassurance(QA)
about/Testingyourapplication
WOW! eBook
www.wowebook.org
R
radix-64
about/Theimports
recursivefunctions
about/Recursivefunctions
Redphase
about/Test-drivendevelopment
Refactor
about/Test-drivendevelopment
regexwebsite
defining/Aregexwebsite
Django,settingup/SettingupDjango
Entrymodel,adding/AddingtheEntrymodel
adminpanel,customizing/Customizingtheadminpanel
form,creating/Creatingtheform
views,writing/Writingtheviews
URLsandviews,using/TyingupURLsandviews
templates,writing/Writingthetemplates
regularexpression/Regularexpressions
relationaldatabase
about/Themodellayer
relativeimports
URL/Relativeimports
request-responseclient-serverprotocol
about/HowdoestheWebwork?
returnvalues
about/Returnvalues
multiplevalues,returning/Returningmultiplevalues
WOW! eBook
www.wowebook.org
S
scheduler
about/Threadingconsiderations
schemamigration
about/AddingtheEntrymodel
Schwartziantransform
about/map
Scikit-Learn
URL/Wheredowegofromhere?
SciPy
URL/Wheredowegofromhere?
scopes
about/Scopes
local/Scopes
enclosing/Scopes
global/Scopes
built-in/Scopes
scopesandnameresolution
defining/Scopesandnameresolution
globalandnonlocalstatements/Theglobalandnonlocalstatements
scripting
about/Firstapproach–scripting
imports/Firstapproach–scripting,Theimports
arguments,parsing/Parsingarguments
businesslogic/Thebusinesslogic
service
Python,runningas/RunningPythonasaservice
service-orientedarchitecture(SOA)
about/Inspectinglogfiles
service-orientedarchitectures
about/Inspectinglogfiles
setter
about/Thepropertydecorator
settypes
about/Settypes
Single-PageApplication(SPA)
about/Thefutureofwebdevelopment
SingleResponsibilityPrinciple(SRP)
about/Writingthehelpers
slicing
about/Aboutindexingandslicing
smallvaluescaching
about/Smallvaluescaching
WOW! eBook
www.wowebook.org
space
about/Someperformanceconsiderations
Sphinx
about/Documentingyourcode
SQL(StructuredQueryLanguage)
about/Themodellayer
staticmethods
about/Staticmethods
statisticalprofiling
about/ProfilingPython
strings
encoding/Encodinganddecodingstrings
decoding/Encodinganddecodingstrings
indexing/Indexingandslicingstrings
slicing/Indexingandslicingstrings
strobjects
about/Stringsandbytes
symmetricencryptionalgorithm
about/Themodellayer
system-exitingexceptions
about/Exceptions
WOW! eBook
www.wowebook.org
T
Tcl(ToolCommandLanguage)/RunningPythonasaGUIapplication
TCP/IP(TransmissionControlProtocol/InternetProtocol)
about/HowdoestheWebwork?
templatelayer
about/Thetemplatelayer
homeandfootertemplates/Homeandfootertemplates
records,listing/Listingallrecords
records,creating/Creatingandeditingrecords
records,editing/Creatingandeditingrecords
API,defining/TalkingtotheAPI
records,deleting/Deletingrecords
terminal
about/SettingupthePythoninterpreter
ternaryoperator
about/Theternaryoperator
test
defining/Theanatomyofatest
preparation/Theanatomyofatest
execution/Theanatomyofatest
verification/Theanatomyofatest
failing/Makingatestfail
test-drivendevelopment(TDD)
about/Test-drivendevelopment
benefits/Test-drivendevelopment
disadvantages/Test-drivendevelopment
testingguidelines
defining/Testingguidelines
tests
front-endtests/Testingyourapplication
scenariotests/Testingyourapplication
integrationtests/Testingyourapplication
smoketests/Testingyourapplication
Acceptancetests/Testingyourapplication
functionaltests/Testingyourapplication
destructivetests/Testingyourapplication
performancetests/Testingyourapplication
usabilitytests/Testingyourapplication
securityandpenetrationtests/Testingyourapplication
unittests/Testingyourapplication
regressiontests/Testingyourapplication
comparing,withmocks/Comparingtestswithandwithoutmocks
comparing,withoutmocks/Comparingtestswithandwithoutmocks
WOW! eBook
www.wowebook.org
boundaries/Boundariesandgranularity
granularity/Boundariesandgranularity
example/Amoreinterestingexample
thread
about/Threadingconsiderations
time
about/Someperformanceconsiderations
Timsort
about/Lists
Tk/RunningPythonasaGUIapplication
tkinter
about/Secondapproach–aGUIapplication
Tkinter/RunningPythonasaGUIapplication
tkinter.tix(TkInterfaceExtension)module
about/Thetkinter.tixmodule
tkinter.tixmodule
about/Thetkinter.tixmodule
Tkinterface
about/Secondapproach–aGUIapplication
triangulation
about/Comparingtestswithandwithoutmocks
troubleshootingguidelines
about/Troubleshootingguidelines
consoleeditors,using/Usingconsoleeditors
inspecting/Wheretoinspect
testsused,fordebugging/Usingteststodebug
monitoring/Monitoring
truedivision(/)
about/Integers
tuple
about/Tuples
turtlemodule
about/Theturtlemodule
WOW! eBook
www.wowebook.org
U
unicodecodepoints
about/Stringsandbytes
UniformResourceLocator(URL)
about/TheDjangoURLdispatcher
unittest
about/Unittesting
writing/Writingaunittest
mockobjects/Mockobjectsandpatching
patching/Mockobjectsandpatching
assertions/Assertions
example/Aclassicunittestexample
unittesting
defining/Unittesting
unpacking
about/Variablepositionalarguments
Upcasting
about/Booleans
useracceptancetesting(UAT)
about/Testingyourapplication
userexperience(UX)
about/Testingyourapplication
Utf-8
about/Encodinganddecodingstrings
WOW! eBook
www.wowebook.org
V
viewlayer
about/Theviewlayer
importsandhomeview/Importsandhomeview
records,listing/Listingallrecords
records,creating/Creatingrecords
records,updating/Updatingrecords
records,deleting/Deletingrecords
views
writing/Writingtheviews
homeview/Thehomeview
entrylistview/Theentrylistview
formview/Theformview
vim
about/Usingconsoleeditors
virtualenv
about/Aboutvirtualenv
referencelink/Aboutvirtualenv
virtualenvironment
creating/Yourfirstvirtualenvironment
WOW! eBook
www.wowebook.org
W
wastedtime
about/Aboutthenames
Web(WorldWideWeb)
defining/WhatistheWeb?
working/HowdoestheWebwork?
webdevelopment
defining/Thefutureofwebdevelopment
webframework
about/TheDjangowebframework
white-boxtests
about/Testingyourapplication
wxPython
about/Secondapproach–aGUIapplication,wxPython,PyQt,andPyGTK
WOW! eBook
www.wowebook.org
Y
yieldfromexpression
about/Theyieldfromexpression
WOW! eBook
www.wowebook.org
Z
zip
about/map,zip,andfilter
defining/zip
WOW! eBook
www.wowebook.org