Python

Packaging

  • python-packaging.readthedocs.io
  • http://www.scotttorborg.com/python-packaging/index.html
  • http://nvie.com/posts/pin-your-packages/
  • http://tech.marksblogg.com/better-python-package-management.html
  • Value error Attempted relative import in non-package

Typing

Internals

Metaclass

  • What's a metaclasse by stackoverflow
  • http://www.slideshare.net/hychen/what-can-meta-class-do-for-you-pycon-taiwan-2012
  • http://www.slideshare.net/gwiener/metaclasses-in-python

TESTS

BEST PRACTICES

  • PEP8: Style Guide for Python Code
  • Design pattern in python
  • dict() vs {} (hint: {} is better)
  • http://excess.org/article/2011/12/unfortunate-python/
  • http://www.canonical.org/~kragen/isinstance/
  • http://www.artima.com/weblogs/viewpost.jsp?thread=236278
  • http://satyajit.ranjeev.in/2012/05/17/python-a-few-things-to-remember.html
  • http://net.tutsplus.com/tutorials/python-tutorials/behavior-driven-development-in-python/
  • Things you didn't know about Python: interesting presentation about Python internal and stuff.
  • Copying list, the right way
  • Make one archive python executable
  • HOWTO Create Python GUIs using HTML
  • Slides about functional versus imperative programming
  • MRO: from official documentation and a post about multiple inheritance (look at also the comments)
  • http://ozkatz.github.com/improving-your-python-productivity.html
  • http://ozkatz.github.com/better-python-apis.html
  • Lazy evaluation
  • https://speakerdeck.com/rwarren/a-brief-intro-to-profiling-in-python
  • http://pyvideo.org/video/1674/getting-started-with-automated-testing
  • http://hynek.me/talks/python-deployments/
  • http://pyrandom.blogspot.nl/2013/04/super-wrong.html
  • Python’s super() considered super!
  • http://www.huyng.com/posts/python-performance-analysis/
  • https://tommikaikkonen.github.io/timezones/
  • format()
  • pyformat.info/

Multithreading&Multiprocessing

Exceptions

LIBRARIES

  • https://github.com/kennethreitz/envoy
  • https://github.com/kennethreitz/requests
  • http://www.nicosphere.net/clint-command-line-library-for-python/
  • Docopts command line arguments parser for Human Beings.
  • Get started with the Natural Language Toolkit
  • pdb++ pdb++, a drop-in replacement for pdb (the Python debugger)
  • napari/napari a fast, interactive, multi-dimensional image viewer for python
  • pydantic Data validation and settings management using python type annotations.
  • pySDR

Scientific

Interesting Stuffs

  • https://jordan-wright.github.io/blog/2014/10/06/creating-tor-hidden-services-with-python/

SANDBOX

  • http://wiki.python.org/moin/Asking%20for%20Help/How%20can%20I%20run%20an%20untrusted%20Python%20script%20safely%20%28i.e.%20Sandbox%29
  • Example of pypy-c-sandbox for launching random scripts
  • http://stackoverflow.com/questions/6655258/using-the-socket-module-in-sandboxed-pypy
  • http://pypy.readthedocs.org/en/latest/sandbox.html
  • http://blog.delroth.net/2013/03/escaping-a-python-sandbox-ndh-2013-quals-writeup/

Instructions for pypy-2.1

$ cd pypy/goal
$ python ../../rpython/bin/rpython  -O2 --sandbox targetpypystandalone.py
$ PYTHONPATH=$PYTHONPATH:$PWD/../../ ../..//pypy/sandbox/pypy_interact.py pypy-c

DEBUG&Profiling

  • Performance analysis
  • CProfile
  • https://stripe.com/blog/exploring-python-using-gdb
  • scalene is a high-performance CPU and memory profiler for Python that does a number of things that other Python profilers do not and cannot do

IDE

  • http://blog.dispatched.ch/2009/05/24/vim-as-python-ide/
$ python -m shlex
kdkd
Token: 'kdkd'
34 5455
Token: '34'
Token: '5455'
$edx=34
Token: '$'
Token: 'edx'
Token: '='
Token: '34'

Time

  • http://www.saltycrane.com/blog/2008/11/python-datetime-time-conversions/
  • http://stackoverflow.com/questions/2775864/python-datetime-to-unix-timestamp

COOKBOOK

>>> a = [1,4,-1,0,13]
>>> a.sort()
>>> a
[-1, 0, 1, 4, 13]
>>> import operator
>>> x = {1: 2, 3: 4, 4:3, 2:1, 0:0}
>>> sorted_x = sorted(x.iteritems(), key=operator.itemgetter(1))

Two's complement

>>> value = 0xb59395a9
>>> f"{ctypes.c_uint32(value).value:032b}"
'10110101100100111001010110101001'
>>> f"{ctypes.c_uint32(~value).value:032b}"
'01001010011011000110101001010110'

Getopt

import getopt, sys

def main():
    try:
        opts, args = getopt.getopt(sys.argv[1:], "ho:v", ["help", "output="])
    except getopt.GetoptError as err:
        # print help information and exit:
        print(err) # will print something like "option -a not recognized"
        usage()
        sys.exit(2)
    output = None
    verbose = False
    for o, a in opts:
        if o == "-v":
            verbose = True
        elif o in ("-h", "--help"):
            usage()
            sys.exit()
        elif o in ("-o", "--output"):
            output = a
        else:
            assert False, "unhandled option"
    # ...

if __name__ == "__main__":
    main()

argparse

def argparse_vendor_product(value):
    vendor, product = tuple(value.split(":"))

    return int(vendor, 16), int(product, 16)


def parse_args():
    args = argparse.ArgumentParser(description='upload and run some code')

    args.add_argument(
        '--device',
        type=argparse_vendor_product,
        required=True,
        help="vendor:product of the device you want to interact with")
    args.add_argument('--binary', required=True)
    args.add_argument('--address', type=functools.partial(int, base=0))

    return args.parse_args()

PySerial

import serial
ser = serial.Serial('/dev/ttyUSB0')  # open serial port
print(ser.name)         # check which port was really used
ser.write(b'hello')     # write a string
ser.close()

Decorator

def trace(f):
    def _inner(*args, **kwargs):
        print ' # ', f.func_name
        return f(*args, **kwargs)
    return _inner

def challenge(count):
    def _challenge(x):
        def _inner(*args, **kwargs):
            print('[+] challenge %d' % count)
            return x(*args, **kwargs)
        return _inner
    return _challenge

DOCTESTS

def decript(cipher, key):
    """
    >>> a = [0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1]
    >>> b = [1, 1, 1, 1, 1, 1, 1]
    >>> decript(a, b) #doctest: +NORMALIZE_WHITESPACE
    [[1, 0, 1, 0, 1, 0, 1],
    [0, 0, 0, 0, 0, 0, 0]]
    """
    r = []
    for i in xrange(0, len(cipher) - len(key) + 1, 7):
        r.append(XOR(cipher[i:i + len(key)], key))

    return r

$ python -m doctest c1.py

SPHINX

It's possible to write the documentation along with the code.

http://sphinx.pocoo.org/markup/toctree.html#toctree-directive

  • https://wiki.python.org/moin/TimeComplexity

Maximum float

Source:

>>> infinity = float("inf")
>>> infinity
inf
>>> infinity / 10000
inf

Print out some docstring for documentation purpose

python -c 'from macro import matrixify;print(matrixify.__doc__.replace("\n    ", "\n"))' | rst2html

Logging

Add a debug level (http://stackoverflow.com/a/16955098/1935366)

  • http://victorlin.me/posts/2012/08/26/good-logging-practice-in-python

Remember that logger.basicConfig() attaches the stream handler by default, if you want to fine tune the logging you have to set it by yourself.

import logging

SUBDEBUG = 5
logging.addLevelName(SUBDEBUG, 'SUBDEBUG')

def subdebug(self, message, *args, **kws):
    self.log(SUBDEBUG, message, *args, **kws) 
logging.Logger.subdebug = subdebug

logging.basicConfig()
l = logging.getLogger()
l.setLevel(SUBDEBUG)
l.subdebug('test')
l.setLevel(logging.DEBUG)
l.subdebug('test')

stream = logging.StreamHandler()
formatter = logging.Formatter('%(levelname)s - %(filename)s:%(lineno)d - %(message)s')

logger = logging.getLogger(__file__)
logger.setLevel(logging.DEBUG)
logger.addHandler(stream)
stream.setFormatter(formatter)

Flatten list

>>> chain = itertools.chain.from_iterable([[1,2],[3],[5,89],[],[6]])
>>> print(list(chain))
>>> [1, 2, 3, 5, 89, 6]

for x in s:
  if x:
      return True
return False

return any(x)

Traceback

    try:
        _manage_object(pk, *args, **kwargs)
    except:
        obj = Object.objects.get(pk=pk)
        # get the exception context to reuse later
        exc_info = sys.exc_info()
        import traceback
        print traceback.print_tb(exc_info[2])

Read/write UTF8 files

Seems like that the builtin open() in python manage only ascii files

import codecs

def create_post(filepath, content):
    with codecs.open(filepath, 'w+', encoding='utf-8') as f:
        f.write(content)

Get first item of a nested list

>>> from operator import itemgetter
>>> rows = [(1, 2), (3, 4), (5, 6)]
>>> map(itemgetter(1), rows)
[2, 4, 6]
>>>

Extract URL from string

import re

myString = "This is my tweet check it out http://tinyurl.com/blah"

print re.search("(?P<url>https?://[^\s]+)", myString).group("url")

Routing from REGEXs

In [1]: import re

In [2]: c = re.compile(r'^w::(?P<type>\w+)::(?P<id>\d*)::')

In [3]: s = 'w::w::1::'

In [5]: m = c.match(s)

In [6]: m.groupdict()
Out[6]: {'id': '1', 'type': 'w'}

Add file into a tarfile from a string

def elaborate_archive(filepath, **kwargs):
    tar_src = tarfile.open(filepath, mode='a')

    version_file = StringIO.StringIO(kwargs['version'])

    version_tarinfo = tarfile.TarInfo('VERSION')
    version_tarinfo.size = len(version_file.buf)
    tar_src.addfile(version_tarinfo, version_file)
    tar_src.close()

pandas

$ pip install pandas

import pandas as pd

You can read data from a CSV

df = pd.read_csv("/path/to/data")

or create manually one

df = pd.DataFrame({
    "column 1": [data1, data2, ..., dataN],
    "column 2": [...],
    ...
})

To have general information about the DataFrame

df.info()

A nice feature is the filtering

df[(df.duration > = 200) & (df.genre == "Drama")]

It's possible to plot directly

df.plot(x='GE', y=['TOTALE_19', 'TOTALE_20'], figsize=(20, 10))