Python
- http://mirnazim.org/writings/python-ecosystem-introduction/
- WTF Python Exploring and understanding Python through surprising snippets
- Code Like a Pythonista: Idiomatic Python
- http://effbot.org/zone/python-with-statement.htm
- http://www.python.org/dev/peps/
- http://www.mindviewinc.com/Books/Python3Patterns/Index.php
- http://stackoverflow.com/questions/986006/python-how-do-i-pass-a-variable-by-reference
- http://agiliq.com/blog/2012/06/understanding-args-and-kwargs/
- http://pythonbooks.revolunet.com/
- http://farmdev.com/talks/unicode/
# -*- coding: utf-8 -*-
- What's main.py
- http://blog.amir.rachum.com/post/39501813266/python-the-dictionary-playbook
- https://developers.google.com/edu/python/
- Vulnerability in session cookie http://vudang.com/2013/01/python-web-framework-from-lfr-to-rce/
- http://blog.amir.rachum.com/post/54770419679/python-common-newbie-mistakes-part-1
- http://blog.amir.rachum.com/post/55024295793/python-common-newbie-mistakes-part-2
- http://pypix.com/tools-and-tips/advanced-regular-expression-tips-techniques/
- http://sebastianraschka.com/Articles/2014_python_performance_tweaks.html
- Parallelism in one line
- http://www.fullstackpython.com/
- Writing Python 2-3 compatible code
- https://github.com/faif/python-patterns
- Decorator cheat sheet
- https://wiki.python.org/moin/TimeComplexity
- Writing sustainable Python scripts
- Why print became a function in python3
- Python 101: iterators, generators, coroutines
- How to Bootstrap a Python Project
- A Practical Guide to Using Setup.py
- Massive memory overhead: Numbers in Python and how NumPy helps
- Reproducible Python Bytecode
- pythontutor
Packaging
- python-packaging.readthedocs.io
- http://www.scotttorborg.com/python-packaging/index.html
- http://nvie.com/posts/pin-your-packages/
- http://tech.marksblogg.com/better-python-package-management.html
- Value error Attempted relative import in non-package
Typing
- typing documentation
- PEP 483 - The Theory of Type Hints
- mypy is an optional static type checker for Python that aims to combine the benefits of dynamic (or "duck") typing and static typin
- python/mypy Optional static typing for Python 3 and 2 (PEP 484)
- Python: better typed than you think mypy assisted error handling, exception mechanisms in other languages, fun with pattern matching and type variance
- Python Typing: Resisting the Any type
Internals
- http://www.jeffknupp.com/blog/2013/02/14/drastically-improve-your-python-understanding-pythons-execution-model/
- Escaping a sandbox using magic of python
- http://stackoverflow.com/questions/878943/why-return-notimplmented-instead-of-raising-notimplementederror
- Using Cython to speed up
- http://late.am/post/2012/03/26/exploring-python-code-objects
- We are all consenting adult here
- Understanding internals of Python classes
- Python Descriptors Demystified
- Python Attribute Access and the Descriptor Protocol
- How python implements long integers?
- open and CPython
- Python behind the scenes #7: how Python attributes work
Metaclass
- What's a metaclasse by stackoverflow
- http://www.slideshare.net/hychen/what-can-meta-class-do-for-you-pycon-taiwan-2012
- http://www.slideshare.net/gwiener/metaclasses-in-python
TESTS
- My Python testing style guide
- https://www.youtube.com/watch?v=wWu_iRuBjKs
- https://www.integralist.co.uk/posts/toxini/
- moto, a library that allows you to easily mock out tests based on AWS infrastructure.
- Testing Python Applications with Pytest
BEST PRACTICES
- PEP8: Style Guide for Python Code
- Design pattern in python
dict()
vs{}
(hint:{}
is better)- http://excess.org/article/2011/12/unfortunate-python/
- http://www.canonical.org/~kragen/isinstance/
- http://www.artima.com/weblogs/viewpost.jsp?thread=236278
- http://satyajit.ranjeev.in/2012/05/17/python-a-few-things-to-remember.html
- http://net.tutsplus.com/tutorials/python-tutorials/behavior-driven-development-in-python/
- Things you didn't know about Python: interesting presentation about Python internal and stuff.
- Copying list, the right way
- Make one archive python executable
- HOWTO Create Python GUIs using HTML
- Slides about functional versus imperative programming
- MRO: from official documentation and a post about multiple inheritance (look at also the comments)
- http://ozkatz.github.com/improving-your-python-productivity.html
- http://ozkatz.github.com/better-python-apis.html
- Lazy evaluation
- https://speakerdeck.com/rwarren/a-brief-intro-to-profiling-in-python
- http://pyvideo.org/video/1674/getting-started-with-automated-testing
- http://hynek.me/talks/python-deployments/
- http://pyrandom.blogspot.nl/2013/04/super-wrong.html
- Python’s super() considered super!
- http://www.huyng.com/posts/python-performance-analysis/
- https://tommikaikkonen.github.io/timezones/
- format()
- pyformat.info/
Multithreading&Multiprocessing
Exceptions
- Exception in python make code clearer, see also this.
- https://julien.danjou.info/blog/2015/python-retrying
- Reraising exception
LIBRARIES
- https://github.com/kennethreitz/envoy
- https://github.com/kennethreitz/requests
- http://www.nicosphere.net/clint-command-line-library-for-python/
- Docopts command line arguments parser for Human Beings.
- Get started with the Natural Language Toolkit
- pdb++ pdb++, a drop-in replacement for pdb (the Python debugger)
- napari/napari a fast, interactive, multi-dimensional image viewer for python
- pydantic Data validation and settings management using python type annotations.
- pySDR
Scientific
Interesting Stuffs
- https://jordan-wright.github.io/blog/2014/10/06/creating-tor-hidden-services-with-python/
SANDBOX
- http://wiki.python.org/moin/Asking%20for%20Help/How%20can%20I%20run%20an%20untrusted%20Python%20script%20safely%20%28i.e.%20Sandbox%29
- Example of pypy-c-sandbox for launching random scripts
- http://stackoverflow.com/questions/6655258/using-the-socket-module-in-sandboxed-pypy
- http://pypy.readthedocs.org/en/latest/sandbox.html
- http://blog.delroth.net/2013/03/escaping-a-python-sandbox-ndh-2013-quals-writeup/
Instructions for pypy-2.1
$ cd pypy/goal
$ python ../../rpython/bin/rpython -O2 --sandbox targetpypystandalone.py
$ PYTHONPATH=$PYTHONPATH:$PWD/../../ ../..//pypy/sandbox/pypy_interact.py pypy-c
DEBUG&Profiling
- Performance analysis
- CProfile
- https://stripe.com/blog/exploring-python-using-gdb
- scalene is a high-performance CPU and memory profiler for Python that does a number of things that other Python profilers do not and cannot do
IDE
- http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/
$ python -m shlex
kdkd
Token: 'kdkd'
34 5455
Token: '34'
Token: '5455'
$edx=34
Token: '$'
Token: 'edx'
Token: '='
Token: '34'
Time
- http://www.saltycrane.com/blog/2008/11/python-datetime-time-conversions/
- http://stackoverflow.com/questions/2775864/python-datetime-to-unix-timestamp
COOKBOOK
>>> a = [1,4,-1,0,13]
>>> a.sort()
>>> a
[-1, 0, 1, 4, 13]
>>> import operator
>>> x = {1: 2, 3: 4, 4:3, 2:1, 0:0}
>>> sorted_x = sorted(x.iteritems(), key=operator.itemgetter(1))
Two's complement
>>> value = 0xb59395a9
>>> f"{ctypes.c_uint32(value).value:032b}"
'10110101100100111001010110101001'
>>> f"{ctypes.c_uint32(~value).value:032b}"
'01001010011011000110101001010110'
Getopt
import getopt, sys
def main():
try:
opts, args = getopt.getopt(sys.argv[1:], "ho:v", ["help", "output="])
except getopt.GetoptError as err:
# print help information and exit:
print(err) # will print something like "option -a not recognized"
usage()
sys.exit(2)
output = None
verbose = False
for o, a in opts:
if o == "-v":
verbose = True
elif o in ("-h", "--help"):
usage()
sys.exit()
elif o in ("-o", "--output"):
output = a
else:
assert False, "unhandled option"
# ...
if __name__ == "__main__":
main()
argparse
def argparse_vendor_product(value):
vendor, product = tuple(value.split(":"))
return int(vendor, 16), int(product, 16)
def parse_args():
args = argparse.ArgumentParser(description='upload and run some code')
args.add_argument(
'--device',
type=argparse_vendor_product,
required=True,
help="vendor:product of the device you want to interact with")
args.add_argument('--binary', required=True)
args.add_argument('--address', type=functools.partial(int, base=0))
return args.parse_args()
PySerial
import serial
ser = serial.Serial('/dev/ttyUSB0') # open serial port
print(ser.name) # check which port was really used
ser.write(b'hello') # write a string
ser.close()
Decorator
def trace(f):
def _inner(*args, **kwargs):
print ' # ', f.func_name
return f(*args, **kwargs)
return _inner
def challenge(count):
def _challenge(x):
def _inner(*args, **kwargs):
print('[+] challenge %d' % count)
return x(*args, **kwargs)
return _inner
return _challenge
DOCTESTS
def decript(cipher, key):
"""
>>> a = [0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1]
>>> b = [1, 1, 1, 1, 1, 1, 1]
>>> decript(a, b) #doctest: +NORMALIZE_WHITESPACE
[[1, 0, 1, 0, 1, 0, 1],
[0, 0, 0, 0, 0, 0, 0]]
"""
r = []
for i in xrange(0, len(cipher) - len(key) + 1, 7):
r.append(XOR(cipher[i:i + len(key)], key))
return r
$ python -m doctest c1.py
SPHINX
It's possible to write the documentation along with the code.
http://sphinx.pocoo.org/markup/toctree.html#toctree-directive
- https://wiki.python.org/moin/TimeComplexity
Maximum float
>>> infinity = float("inf")
>>> infinity
inf
>>> infinity / 10000
inf
Print out some docstring for documentation purpose
python -c 'from macro import matrixify;print(matrixify.__doc__.replace("\n ", "\n"))' | rst2html
Logging
- Documentation
- http://victorlin.me/2012/08/good-logging-practice-in-python/
- http://hynek.me/articles/taking-some-pain-out-of-python-logging/
- Multi line formatting
Add a debug level (http://stackoverflow.com/a/16955098/1935366)
- http://victorlin.me/posts/2012/08/26/good-logging-practice-in-python
Remember that logger.basicConfig()
attaches the stream handler by default, if you want
to fine tune the logging you have to set it by yourself.
import logging
SUBDEBUG = 5
logging.addLevelName(SUBDEBUG, 'SUBDEBUG')
def subdebug(self, message, *args, **kws):
self.log(SUBDEBUG, message, *args, **kws)
logging.Logger.subdebug = subdebug
logging.basicConfig()
l = logging.getLogger()
l.setLevel(SUBDEBUG)
l.subdebug('test')
l.setLevel(logging.DEBUG)
l.subdebug('test')
stream = logging.StreamHandler()
formatter = logging.Formatter('%(levelname)s - %(filename)s:%(lineno)d - %(message)s')
logger = logging.getLogger(__file__)
logger.setLevel(logging.DEBUG)
logger.addHandler(stream)
stream.setFormatter(formatter)
Flatten list
>>> chain = itertools.chain.from_iterable([[1,2],[3],[5,89],[],[6]])
>>> print(list(chain))
>>> [1, 2, 3, 5, 89, 6]
for x in s:
if x:
return True
return False
return any(x)
Traceback
try:
_manage_object(pk, *args, **kwargs)
except:
obj = Object.objects.get(pk=pk)
# get the exception context to reuse later
exc_info = sys.exc_info()
import traceback
print traceback.print_tb(exc_info[2])
Read/write UTF8 files
Seems like that the builtin open()
in python manage only ascii
files
import codecs
def create_post(filepath, content):
with codecs.open(filepath, 'w+', encoding='utf-8') as f:
f.write(content)
Get first item of a nested list
>>> from operator import itemgetter
>>> rows = [(1, 2), (3, 4), (5, 6)]
>>> map(itemgetter(1), rows)
[2, 4, 6]
>>>
Extract URL from string
import re
myString = "This is my tweet check it out http://tinyurl.com/blah"
print re.search("(?P<url>https?://[^\s]+)", myString).group("url")
Routing from REGEXs
In [1]: import re
In [2]: c = re.compile(r'^w::(?P<type>\w+)::(?P<id>\d*)::')
In [3]: s = 'w::w::1::'
In [5]: m = c.match(s)
In [6]: m.groupdict()
Out[6]: {'id': '1', 'type': 'w'}
Add file into a tarfile from a string
def elaborate_archive(filepath, **kwargs):
tar_src = tarfile.open(filepath, mode='a')
version_file = StringIO.StringIO(kwargs['version'])
version_tarinfo = tarfile.TarInfo('VERSION')
version_tarinfo.size = len(version_file.buf)
tar_src.addfile(version_tarinfo, version_file)
tar_src.close()
pandas
$ pip install pandas
import pandas as pd
You can read data from a CSV
df = pd.read_csv("/path/to/data")
or create manually one
df = pd.DataFrame({
"column 1": [data1, data2, ..., dataN],
"column 2": [...],
...
})
To have general information about the DataFrame
df.info()
A nice feature is the filtering
df[(df.duration > = 200) & (df.genre == "Drama")]
It's possible to plot directly
df.plot(x='GE', y=['TOTALE_19', 'TOTALE_20'], figsize=(20, 10))