Inside python interpreter

tags:`python`

Python 3.5

default builtins

sys.modules is a dictionary mapping the names of imported modules to the module object holding the code

import sys
dir(sys.modules['builtins']) == dir(__builtins__)

True

sorted(sys.modules.keys())

['IPython',
 'IPython.core',
 'IPython.core.alias',
 'IPython.core.application',
 'IPython.core.autocall',
 'IPython.core.builtin_trap',
 'IPython.core.compilerop',
 'IPython.core.completer',
 'IPython.core.completerlib',
 'IPython.core.crashhandler',
 'IPython.core.debugger',
 'IPython.core.display',
 'IPython.core.display_trap',
 'IPython.core.displayhook',
 'IPython.core.displaypub',
 'IPython.core.error',
 'IPython.core.events',
 'IPython.core.excolors',
 'IPython.core.extensions',
 'IPython.core.formatters',
 'IPython.core.getipython',
 'IPython.core.history',
 'IPython.core.hooks',
 'IPython.core.inputsplitter',
 'IPython.core.inputtransformer',
 'IPython.core.interactiveshell',
 'IPython.core.latex_symbols',
 'IPython.core.logger',
 'IPython.core.macro',
 'IPython.core.magic',
 'IPython.core.magic_arguments',
 'IPython.core.magics',
 'IPython.core.magics.auto',
 'IPython.core.magics.basic',
 'IPython.core.magics.code',
 'IPython.core.magics.config',
 'IPython.core.magics.display',
 'IPython.core.magics.execution',
 'IPython.core.magics.extension',
 'IPython.core.magics.history',
 'IPython.core.magics.logging',
 'IPython.core.magics.namespace',
 'IPython.core.magics.osm',
 'IPython.core.magics.pylab',
 'IPython.core.magics.script',
 'IPython.core.oinspect',
 'IPython.core.page',
 'IPython.core.payload',
 'IPython.core.payloadpage',
 'IPython.core.prefilter',
 'IPython.core.profiledir',
 'IPython.core.pylabtools',
 'IPython.core.release',
 'IPython.core.shellapp',
 'IPython.core.splitinput',
 'IPython.core.ultratb',
 'IPython.core.usage',
 'IPython.display',
 'IPython.extensions',
 'IPython.extensions.storemagic',
 'IPython.lib',
 'IPython.lib.backgroundjobs',
 'IPython.lib.clipboard',
 'IPython.lib.display',
 'IPython.lib.pretty',
 'IPython.lib.security',
 'IPython.paths',
 'IPython.terminal',
 'IPython.terminal.debugger',
 'IPython.terminal.embed',
 'IPython.terminal.interactiveshell',
 'IPython.terminal.ipapp',
 'IPython.terminal.magics',
 'IPython.terminal.prompts',
 'IPython.terminal.pt_inputhooks',
 'IPython.terminal.ptutils',
 'IPython.terminal.shortcuts',
 'IPython.testing',
 'IPython.testing.skipdoctest',
 'IPython.utils',
 'IPython.utils.PyColorize',
 'IPython.utils._process_common',
 'IPython.utils._process_posix',
 'IPython.utils._sysinfo',
 'IPython.utils.capture',
 'IPython.utils.colorable',
 'IPython.utils.coloransi',
 'IPython.utils.contexts',
 'IPython.utils.data',
 'IPython.utils.decorators',
 'IPython.utils.dir2',
 'IPython.utils.encoding',
 'IPython.utils.frame',
 'IPython.utils.generics',
 'IPython.utils.importstring',
 'IPython.utils.io',
 'IPython.utils.ipstruct',
 'IPython.utils.module_paths',
 'IPython.utils.openpy',
 'IPython.utils.path',
 'IPython.utils.process',
 'IPython.utils.py3compat',
 'IPython.utils.sentinel',
 'IPython.utils.strdispatch',
 'IPython.utils.sysinfo',
 'IPython.utils.syspathcontext',
 'IPython.utils.tempdir',
 'IPython.utils.terminal',
 'IPython.utils.text',
 'IPython.utils.timing',
 'IPython.utils.tokenize2',
 'IPython.utils.tokenutil',
 'IPython.utils.wildcard',
 '__future__',
 '__main__',
 '__mp_main__',
 '_ast',
 '_bisect',
 '_bootlocale',
 '_bz2',
 '_codecs',
 '_collections',
 '_collections_abc',
 '_compat_pickle',
 '_compression',
 '_ctypes',
 '_curses',
 '_cython_0_25_1',
 '_datetime',
 '_decimal',
 '_frozen_importlib',
 '_frozen_importlib_external',
 '_functools',
 '_hashlib',
 '_heapq',
 '_imp',
 '_io',
 '_json',
 '_locale',
 '_lsprof',
 '_lzma',
 '_multiprocessing',
 '_opcode',
 '_operator',
 '_pickle',
 '_posixsubprocess',
 '_random',
 '_signal',
 '_sitebuiltins',
 '_socket',
 '_sqlite3',
 '_sre',
 '_stat',
 '_string',
 '_strptime',
 '_struct',
 '_sysconfigdata',
 '_sysconfigdata_m',
 '_thread',
 '_warnings',
 '_weakref',
 '_weakrefset',
 'abc',
 'apport_python_hook',
 'argparse',
 'array',
 'ast',
 'atexit',
 'base64',
 'bdb',
 'binascii',
 'bisect',
 'builtins',
 'bz2',
 'cProfile',
 'calendar',
 'cmd',
 'code',
 'codecs',
 'codeop',
 'collections',
 'collections.abc',
 'concurrent',
 'concurrent.futures',
 'concurrent.futures._base',
 'concurrent.futures.process',
 'concurrent.futures.thread',
 'contextlib',
 'copy',
 'copyreg',
 'ctypes',
 'ctypes._endian',
 'ctypes.util',
 'curses',
 'datetime',
 'dateutil',
 'dateutil._common',
 'dateutil.parser',
 'dateutil.relativedelta',
 'dateutil.tz',
 'dateutil.tz._common',
 'dateutil.tz.tz',
 'decimal',
 'decorator',
 'difflib',
 'dis',
 'email',
 'email._encoded_words',
 'email._parseaddr',
 'email._policybase',
 'email.base64mime',
 'email.charset',
 'email.encoders',
 'email.errors',
 'email.feedparser',
 'email.header',
 'email.iterators',
 'email.message',
 'email.parser',
 'email.quoprimime',
 'email.utils',
 'encodings',
 'encodings.aliases',
 'encodings.latin_1',
 'encodings.utf_8',
 'enum',
 'errno',
 'faulthandler',
 'fcntl',
 'fnmatch',
 'functools',
 'gc',
 'genericpath',
 'getopt',
 'getpass',
 'gettext',
 'glob',
 'grp',
 'hashlib',
 'heapq',
 'hmac',
 'html',
 'html.entities',
 'imp',
 'importlib',
 'importlib._bootstrap',
 'importlib._bootstrap_external',
 'importlib.abc',
 'importlib.machinery',
 'importlib.util',
 'inspect',
 'io',
 'ipykernel',
 'ipykernel._version',
 'ipykernel.codeutil',
 'ipykernel.comm',
 'ipykernel.comm.comm',
 'ipykernel.comm.manager',
 'ipykernel.connect',
 'ipykernel.datapub',
 'ipykernel.displayhook',
 'ipykernel.heartbeat',
 'ipykernel.iostream',
 'ipykernel.ipkernel',
 'ipykernel.jsonutil',
 'ipykernel.kernelapp',
 'ipykernel.kernelbase',
 'ipykernel.parentpoller',
 'ipykernel.pickleutil',
 'ipykernel.serialize',
 'ipykernel.zmqshell',
 'ipython_genutils',
 'ipython_genutils._version',
 'ipython_genutils.encoding',
 'ipython_genutils.importstring',
 'ipython_genutils.path',
 'ipython_genutils.py3compat',
 'ipython_genutils.text',
 'ipywidgets',
 'ipywidgets._version',
 'ipywidgets.widgets',
 'ipywidgets.widgets.domwidget',
 'ipywidgets.widgets.interaction',
 'ipywidgets.widgets.trait_types',
 'ipywidgets.widgets.valuewidget',
 'ipywidgets.widgets.widget',
 'ipywidgets.widgets.widget_bool',
 'ipywidgets.widgets.widget_box',
 'ipywidgets.widgets.widget_button',
 'ipywidgets.widgets.widget_color',
 'ipywidgets.widgets.widget_controller',
 'ipywidgets.widgets.widget_core',
 'ipywidgets.widgets.widget_date',
 'ipywidgets.widgets.widget_float',
 'ipywidgets.widgets.widget_image',
 'ipywidgets.widgets.widget_int',
 'ipywidgets.widgets.widget_layout',
 'ipywidgets.widgets.widget_link',
 'ipywidgets.widgets.widget_output',
 'ipywidgets.widgets.widget_selection',
 'ipywidgets.widgets.widget_selectioncontainer',
 'ipywidgets.widgets.widget_string',
 'ipywidgets.widgets.widget_style',
 'itertools',
 'jedi',
 'jedi._compatibility',
 'jedi.api',
 'jedi.api.classes',
 'jedi.api.completion',
 'jedi.api.helpers',
 'jedi.api.interpreter',
 'jedi.api.keywords',
 'jedi.api.usages',
 'jedi.cache',
 'jedi.common',
 'jedi.debug',
 'jedi.evaluate',
 'jedi.evaluate.analysis',
 'jedi.evaluate.cache',
 'jedi.evaluate.compiled',
 'jedi.evaluate.compiled.fake',
 'jedi.evaluate.compiled.mixed',
 'jedi.evaluate.context',
 'jedi.evaluate.docstrings',
 'jedi.evaluate.dynamic',
 'jedi.evaluate.filters',
 'jedi.evaluate.finder',
 'jedi.evaluate.flow_analysis',
 'jedi.evaluate.helpers',
 'jedi.evaluate.imports',
 'jedi.evaluate.instance',
 'jedi.evaluate.iterable',
 'jedi.evaluate.param',
 'jedi.evaluate.pep0484',
 'jedi.evaluate.precedence',
 'jedi.evaluate.recursion',
 'jedi.evaluate.representation',
 'jedi.evaluate.site',
 'jedi.evaluate.stdlib',
 'jedi.evaluate.sys_path',
 'jedi.parser',
 'jedi.parser.cache',
 'jedi.parser.parser',
 'jedi.parser.pgen2',
 'jedi.parser.pgen2.grammar',
 'jedi.parser.pgen2.parse',
 'jedi.parser.pgen2.pgen',
 'jedi.parser.python',
 'jedi.parser.python.diff',
 'jedi.parser.python.parser',
 'jedi.parser.python.tree',
 'jedi.parser.token',
 'jedi.parser.tokenize',
 'jedi.parser.tree',
 'jedi.settings',
 'json',
 'json.decoder',
 'json.encoder',
 'json.scanner',
 'jupyter_client',
 'jupyter_client._version',
 'jupyter_client.adapter',
 'jupyter_client.blocking',
 'jupyter_client.blocking.channels',
 'jupyter_client.blocking.client',
 'jupyter_client.channels',
 'jupyter_client.channelsabc',
 'jupyter_client.client',
 'jupyter_client.clientabc',
 'jupyter_client.connect',
 'jupyter_client.jsonutil',
 'jupyter_client.kernelspec',
 'jupyter_client.launcher',
 'jupyter_client.localinterfaces',
 'jupyter_client.manager',
 'jupyter_client.managerabc',
 'jupyter_client.multikernelmanager',
 'jupyter_client.session',
 'jupyter_core',
 'jupyter_core.paths',
 'jupyter_core.version',
 'keyrings',
 'keyword',
 'linecache',
 'locale',
 'logging',
 'logging.handlers',
 'lzma',
 'marshal',
 'math',
 'mimetypes',
 'multiprocessing',
 'multiprocessing.connection',
 'multiprocessing.context',
 'multiprocessing.process',
 'multiprocessing.reduction',
 'multiprocessing.util',
 'ntpath',
 'numbers',
 'opcode',
 'operator',
 'optparse',
 'os',
 'os.path',
 'pathlib',
 'pdb',
 'pexpect',
 'pexpect.exceptions',
 'pexpect.expect',
 'pexpect.pty_spawn',
 'pexpect.run',
 'pexpect.spawnbase',
 'pexpect.utils',
 'pickle',
 'pickleshare',
 'pkg_resources',
 'pkg_resources._vendor',
 'pkg_resources._vendor.packaging.__about__',
 'pkg_resources._vendor.six',
 'pkg_resources._vendor.six.moves',
 'pkg_resources.extern',
 'pkg_resources.extern.appdirs',
 'pkg_resources.extern.packaging',
 'pkg_resources.extern.packaging._compat',
 'pkg_resources.extern.packaging._structures',
 'pkg_resources.extern.packaging.markers',
 'pkg_resources.extern.packaging.requirements',
 'pkg_resources.extern.packaging.specifiers',
 'pkg_resources.extern.packaging.version',
 'pkg_resources.extern.pyparsing',
 'pkg_resources.extern.six',
 'pkg_resources.extern.six.moves',
 'pkg_resources.extern.six.moves.urllib',
 'pkgutil',
 'platform',
 'plistlib',
 'posix',
 'posixpath',
 'pprint',
 'profile',
 'prompt_toolkit',
 'prompt_toolkit.application',
 'prompt_toolkit.auto_suggest',
 'prompt_toolkit.buffer',
 'prompt_toolkit.buffer_mapping',
 'prompt_toolkit.cache',
 'prompt_toolkit.clipboard',
 'prompt_toolkit.clipboard.base',
 'prompt_toolkit.clipboard.in_memory',
 'prompt_toolkit.completion',
 'prompt_toolkit.document',
 'prompt_toolkit.enums',
 'prompt_toolkit.eventloop',
 'prompt_toolkit.eventloop.base',
 'prompt_toolkit.eventloop.callbacks',
 'prompt_toolkit.filters',
 'prompt_toolkit.filters.base',
 'prompt_toolkit.filters.cli',
 'prompt_toolkit.filters.types',
 'prompt_toolkit.filters.utils',
 'prompt_toolkit.history',
 'prompt_toolkit.input',
 'prompt_toolkit.interface',
 'prompt_toolkit.key_binding',
 'prompt_toolkit.key_binding.bindings',
 'prompt_toolkit.key_binding.bindings.basic',
 'prompt_toolkit.key_binding.bindings.completion',
 'prompt_toolkit.key_binding.bindings.emacs',
 'prompt_toolkit.key_binding.bindings.named_commands',
 'prompt_toolkit.key_binding.bindings.scroll',
 'prompt_toolkit.key_binding.bindings.vi',
 'prompt_toolkit.key_binding.defaults',
 'prompt_toolkit.key_binding.digraphs',
 'prompt_toolkit.key_binding.input_processor',
 'prompt_toolkit.key_binding.manager',
 'prompt_toolkit.key_binding.registry',
 'prompt_toolkit.key_binding.vi_state',
 'prompt_toolkit.keys',
 'prompt_toolkit.layout',
 'prompt_toolkit.layout.containers',
 'prompt_toolkit.layout.controls',
 'prompt_toolkit.layout.dimension',
 'prompt_toolkit.layout.lexers',
 'prompt_toolkit.layout.margins',
 'prompt_toolkit.layout.menus',
 'prompt_toolkit.layout.mouse_handlers',
 'prompt_toolkit.layout.processors',
 'prompt_toolkit.layout.prompt',
 'prompt_toolkit.layout.screen',
 'prompt_toolkit.layout.toolbars',
 'prompt_toolkit.layout.utils',
 'prompt_toolkit.mouse_events',
 'prompt_toolkit.output',
 'prompt_toolkit.reactive',
 'prompt_toolkit.renderer',
 'prompt_toolkit.search_state',
 'prompt_toolkit.selection',
 'prompt_toolkit.shortcuts',
 'prompt_toolkit.styles',
 'prompt_toolkit.styles.base',
 'prompt_toolkit.styles.defaults',
 'prompt_toolkit.styles.from_dict',
 'prompt_toolkit.styles.from_pygments',
 'prompt_toolkit.styles.utils',
 'prompt_toolkit.terminal',
 'prompt_toolkit.terminal.vt100_input',
 'prompt_toolkit.terminal.vt100_output',
 'prompt_toolkit.token',
 'prompt_toolkit.utils',
 'prompt_toolkit.validation',
 'pstats',
 'pty',
 'ptyprocess',
 'ptyprocess.ptyprocess',
 'ptyprocess.util',
 'pwd',
 'pydoc',
 'pydoc_data',
 'pydoc_data.topics',
 'pyexpat',
 'pyexpat.errors',
 'pyexpat.model',
 'pygments',
 'pygments.filter',
 'pygments.filters',
 'pygments.formatter',
 'pygments.formatters',
 'pygments.formatters._mapping',
 'pygments.formatters.html',
 'pygments.lexer',
 'pygments.lexers',
 'pygments.lexers._mapping',
 'pygments.lexers.python',
 'pygments.modeline',
 'pygments.plugin',
 'pygments.regexopt',
 'pygments.style',
 'pygments.styles',
 'pygments.styles.default',
 'pygments.token',
 'pygments.unistring',
 'pygments.util',
 'queue',
 'quopri',
 'random',
 're',
 'reprlib',
 'resource',
 'runpy',
 'select',
 'selectors',
 'shlex',
 'shutil',
 'signal',
 'simplegeneric',
 'simplejson',
 'simplejson._speedups',
 'simplejson.compat',
 'simplejson.decoder',
 'simplejson.encoder',
 'simplejson.scanner',
 'site',
 'sitecustomize',
 'six',
 'six.moves',
 'socket',
 'sqlite3',
 'sqlite3.dbapi2',
 'sre_compile',
 'sre_constants',
 'sre_parse',
 'stat',
 'storemagic',
 'string',
 'struct',
 'subprocess',
 'sys',
 'sysconfig',
 'tempfile',
 'termios',
 'textwrap',
 'threading',
 'time',
 'timeit',
 'token',
 'tokenize',
 'tornado',
 'tornado.concurrent',
 'tornado.escape',
 'tornado.ioloop',
 'tornado.log',
 'tornado.platform',
 'tornado.platform.auto',
 'tornado.platform.common',
 'tornado.platform.interface',
 'tornado.platform.posix',
 'tornado.speedups',
 'tornado.stack_context',
 'tornado.util',
 'traceback',
 'traitlets',
 'traitlets._version',
 'traitlets.config',
 'traitlets.config.application',
 'traitlets.config.configurable',
 'traitlets.config.loader',
 'traitlets.log',
 'traitlets.traitlets',
 'traitlets.utils',
 'traitlets.utils.bunch',
 'traitlets.utils.getargspec',
 'traitlets.utils.importstring',
 'traitlets.utils.sentinel',
 'tty',
 'types',
 'typing',
 'typing.io',
 'typing.re',
 'unicodedata',
 'urllib',
 'urllib.parse',
 'uu',
 'uuid',
 'warnings',
 'wcwidth',
 'wcwidth.table_wide',
 'wcwidth.table_zero',
 'wcwidth.wcwidth',
 'weakref',
 'xml',
 'xml.parsers',
 'xml.parsers.expat',
 'xml.parsers.expat.errors',
 'xml.parsers.expat.model',
 'zipfile',
 'zipimport',
 'zlib',
 'zmq',
 'zmq.backend',
 'zmq.backend.cython',
 'zmq.backend.cython._device',
 'zmq.backend.cython._poll',
 'zmq.backend.cython._version',
 'zmq.backend.cython.constants',
 'zmq.backend.cython.context',
 'zmq.backend.cython.error',
 'zmq.backend.cython.message',
 'zmq.backend.cython.socket',
 'zmq.backend.cython.utils',
 'zmq.backend.select',
 'zmq.error',
 'zmq.eventloop',
 'zmq.eventloop.ioloop',
 'zmq.eventloop.zmqstream',
 'zmq.sugar',
 'zmq.sugar.attrsettr',
 'zmq.sugar.constants',
 'zmq.sugar.context',
 'zmq.sugar.frame',
 'zmq.sugar.poll',
 'zmq.sugar.socket',
 'zmq.sugar.stopwatch',
 'zmq.sugar.tracker',
 'zmq.sugar.version',
 'zmq.utils',
 'zmq.utils.constant_names',
 'zmq.utils.jsonapi',
 'zmq.utils.strtypes']

Built-in Modules

for name in sys.builtin_module_names:
    print(name)

_ast
_bisect
_codecs
_collections
_datetime
_elementtree
_functools
_heapq
_imp
_io
_locale
_md5
_operator
_pickle
_posixsubprocess
_random
_sha1
_sha256
_sha512
_signal
_socket
_sre
_stat
_string
_struct
_symtable
_thread
_tracemalloc
_warnings
_weakref
array
atexit
binascii
builtins
errno
faulthandler
fcntl
gc
grp
itertools
marshal
math
posix
pwd
pyexpat
select
spwd
sys
syslog
time
unicodedata
xxsubtype
zipimport
zlib

Import Path

import sys

for d in sys.path:
    print(d)

/usr/lib/python35.zip
/usr/lib/python3.5
/usr/lib/python3.5/plat-x86_64-linux-gnu
/usr/lib/python3.5/lib-dynload
/home/splasky/.local/lib/python3.5/site-packages
/usr/local/lib/python3.5/dist-packages
/usr/lib/python3/dist-packages
/usr/local/lib/python3.5/dist-packages/IPython/extensions
/home/splasky/.ipython

%%writefile example.py
DATA='a'

Writing example.py

import example

print(example.__file__)
print(example.DATA)

/home/splasky/jupyter-notebook/example.py
a

Changing the path between the initial import and the call to reload() means a different module may be loaded the second time.

Import hook

import sys

class NoisyImportFinder(object):
    
    PATH_TRIGGER = 'NoisyImportFinder_PATH_TRIGGER'
    
    def __init__(self, path_entry):
        print ('Checking NoisyImportFinder support for %s' % path_entry)
        if path_entry != self.PATH_TRIGGER:
            print ('NoisyImportFinder does not work for %s' % path_entry)
            raise ImportError()
        return
    
    def find_module(self, fullname, path=None):
        print ('NoisyImportFinder looking for "%s"' % fullname)
        return None

sys.path_hooks.append(NoisyImportFinder)

sys.path.insert(0, NoisyImportFinder.PATH_TRIGGER)

try:
    import target_module
except Exception as e:
    print ('Import failed:', e)

Checking NoisyImportFinder support for NoisyImportFinder_PATH_TRIGGER
NoisyImportFinder looking for "target_module"
Import failed: No module named 'target_module'

inside the main

dir(sys.modules['__main__'])

['In', 'NamespaceMagics', 'NoisyImportFinder', 'Out', '_', '_10', '_7', '_Jupyter', '__', '___', '__builtin__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '_dh', '_getsizeof', '_i', '_i1', '_i10', '_i11', '_i12', '_i13', '_i14', '_i15', '_i16', '_i17', '_i18', '_i19', '_i2', '_i20', '_i21', '_i22', '_i23', '_i24', '_i3', '_i4', '_i5', '_i6', '_i7', '_i8', '_i9', '_ih', '_ii', '_iii', '_nms', '_oh', 'd', 'dis', 'example', 'exit', 'get_ipython', 'getsizeof', 'json', 'name', 'oo', 'pprint', 'print_function', 'quit', 'sys', 'var_dic_list']

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

How python import work?

import dis

dis.dis("import sys")

  1           0 LOAD_CONST               0 (0)
              3 LOAD_CONST               1 (None)
              6 IMPORT_NAME              0 (sys)
              9 STORE_NAME               0 (sys)
             12 LOAD_CONST               1 (None)
             15 RETURN_VALUE

dis.dis("import os")

  1           0 LOAD_CONST               0 (0)
              3 LOAD_CONST               1 (None)
              6 IMPORT_NAME              0 (os)
              9 STORE_NAME               0 (os)
             12 LOAD_CONST               1 (None)
             15 RETURN_VALUE

# import module dynamically
sys = __import__('sys') 
print(sys)

<module 'sys' (built-in)>

oo = __import__('os') 
print(oo)

<module 'os' from '/usr/lib/python3.5/os.py'>

%%writefile c_modules.c
#include <Python.h>

static PyObject* helloworld(PyObject* self)
{
    return Py_BuildValue("s", "Hello, Python extensions!!");
}

static char helloworld_docs[] =
    "helloworld( ): Any message you want to put here!!\n";

static PyMethodDef helloworld_funcs[] = {
    {"helloworld", helloworld, 
     METH_NOARGS, helloworld_docs},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef helloworldModule = {
    PyModuleDef_HEAD_INIT,
    "hello",   /* name of module */
    helloworld_docs, /* module documentation, may be NULL */
    -1,       /* size of per-interpreter state of the module,
                 or -1 if the module keeps state in global variables. */
    helloworld_funcs
};


PyMODINIT_FUNC
PyInit_helloworld(void)
{
    return PyModule_Create(&helloworldModule);
}

Overwriting c_modules.c

%%writefile setup.py

from distutils.core import setup, Extension
setup(name='helloworld', version='1.0',  \
      ext_modules=[Extension('helloworld', ['c_modules.c'])])

Overwriting setup.py

python3 ./setup.py install

look inside how c module import into python

# dis_import.py
import helloworld

helloworld.helloworld()

dis assembly the module

python -m dis dis_import.py

  6           0 LOAD_CONST               0 (0)
              2 LOAD_CONST               1 (None)
              4 IMPORT_NAME              0 (helloworld)
              6 STORE_NAME               0 (helloworld)

  8           8 LOAD_NAME                0 (helloworld)
             10 LOAD_METHOD              0 (helloworld)
             12 CALL_METHOD              0
             14 POP_TOP
             16 LOAD_CONST               1 (None)
             18 RETURN_VALUE

as same as like import python module

take more look inside source code

IMPORT_NAME:

 TARGET(IMPORT_NAME) {                                                                                                                                              
      ¦   ¦   PyObject *name = GETITEM(names, oparg);
      ¦   ¦   PyObject *fromlist = POP();
      ¦   ¦   PyObject *level = TOP();
      ¦   ¦   PyObject *res;
      ¦   ¦   res = import_name(f, name, fromlist, level);
      ¦   ¦   Py_DECREF(level);
      ¦   ¦   Py_DECREF(fromlist);
      ¦   ¦   SET_TOP(res);
      ¦   ¦   if (res == NULL)
      ¦   ¦   ¦   goto error;
      ¦   ¦   DISPATCH();
      ¦   }

...

static PyObject *
  import_name(PyFrameObject *f, PyObject *name, PyObject *fromlist, PyObject *level)
  {   
      _Py_IDENTIFIER(__import__);
      PyObject *import_func, *res;
      PyObject* stack[5];
      
      import_func = _PyDict_GetItemId(f->f_builtins, &PyId___import__);
      if (import_func == NULL) {
      ¦   PyErr_SetString(PyExc_ImportError, "__import__ not found");
      ¦   return NULL;
      }
      
      /* Fast path for not overloaded __import__. */
      if (import_func == PyThreadState_GET()->interp->import_func) {
      ¦   int ilevel = _PyLong_AsInt(level);
      ¦   if (ilevel == -1 && PyErr_Occurred()) {
      ¦   ¦   return NULL;
      ¦   }
      ¦   res = PyImport_ImportModuleLevelObject(
      ¦   ¦   ¦   ¦   ¦   name,
      ¦   ¦   ¦   ¦   ¦   f->f_globals,
      ¦   ¦   ¦   ¦   ¦   f->f_locals == NULL ? Py_None : f->f_locals,
      ¦   ¦   ¦   ¦   ¦   fromlist,
      ¦   ¦   ¦   ¦   ¦   ilevel);
      ¦   return res;
      }
  
      Py_INCREF(import_func);
      
      stack[0] = name;
      stack[1] = f->f_globals;
      stack[2] = f->f_locals == NULL ? Py_None : f->f_locals;
      stack[3] = fromlist;
      stack[4] = level;
      res = _PyObject_FastCall(import_func, stack, 5);
      Py_DECREF(import_func);
      return res;
}

PyObject *
  PyImport_ImportModuleLevelObject(PyObject *name, PyObject *globals,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦PyObject *locals, PyObject *fromlist,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦int level)
  {
      _Py_IDENTIFIER(_find_and_load);
      _Py_IDENTIFIER(_handle_fromlist);
      PyObject *abs_name = NULL;
      PyObject *final_mod = NULL;
      PyObject *mod = NULL;
      PyObject *package = NULL;
      PyInterpreterState *interp = PyThreadState_GET()->interp;
      int has_from;
  
      if (name == NULL) {
      ¦   PyErr_SetString(PyExc_ValueError, "Empty module name");
      ¦   goto error;
      }
  
      /* The below code is importlib.__import__() & _gcd_import(), ported to C
      ¦  for added performance. */
  
      if (!PyUnicode_Check(name)) {
      ¦   PyErr_SetString(PyExc_TypeError, "module name must be a string");
      ¦   goto error;
      }
      if (PyUnicode_READY(name) < 0) {
      ¦   goto error;
      }
      if (level < 0) {
      ¦   PyErr_SetString(PyExc_ValueError, "level must be >= 0");
      ¦   goto error;
      }
  
      if (level > 0) {
      ¦   abs_name = resolve_name(name, globals, level);
      ¦   if (abs_name == NULL)
      ¦   ¦   goto error;
      }                        
      else {  /* level == 0 */
      ¦   if (PyUnicode_GET_LENGTH(name) == 0) {
      ¦   ¦   PyErr_SetString(PyExc_ValueError, "Empty module name");
      ¦   ¦   goto error;
      ¦   }
      ¦   abs_name = name;
      ¦   Py_INCREF(abs_name);
      }
  
      mod = PyImport_GetModule(abs_name);
      if (mod != NULL && mod != Py_None) {
      ¦   _Py_IDENTIFIER(__spec__);
      ¦   _Py_IDENTIFIER(_initializing);
      ¦   _Py_IDENTIFIER(_lock_unlock_module);
      ¦   PyObject *value = NULL;
      ¦   PyObject *spec;
      ¦   int initializing = 0;
  
      ¦   /* Optimization: only call _bootstrap._lock_unlock_module() if
      ¦   ¦  __spec__._initializing is true.
      ¦   ¦  NOTE: because of this, initializing must be set *before*
      ¦   ¦  stuffing the new module in sys.modules.
      ¦   ¦*/
      ¦   spec = _PyObject_GetAttrId(mod, &PyId___spec__);
      ¦   if (spec != NULL) {
      ¦   ¦   value = _PyObject_GetAttrId(spec, &PyId__initializing);
      ¦   ¦   Py_DECREF(spec);
      ¦   }
      ¦   if (value == NULL)
      ¦   ¦   PyErr_Clear();
      ¦   else {
      ¦   ¦   initializing = PyObject_IsTrue(value);
      ¦   ¦   Py_DECREF(value);
      ¦   ¦   if (initializing == -1)
      ¦   ¦   ¦   PyErr_Clear();
      ¦   ¦   if (initializing > 0) {
      ¦   ¦   ¦   value = _PyObject_CallMethodIdObjArgs(interp->importlib,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   &PyId__lock_unlock_module, abs_name,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   NULL);  
              ¦   if (value == NULL)
      ¦   ¦   ¦   ¦   goto error;
      ¦   ¦   ¦   Py_DECREF(value);
      ¦   ¦   }
      ¦   }
      }
      else {
      ¦   static int ximporttime = 0;
      ¦   static int import_level;
      ¦   static _PyTime_t accumulated;
      ¦   _Py_IDENTIFIER(importtime);
  
      ¦   _PyTime_t t1 = 0, accumulated_copy = accumulated;
  
      ¦   Py_XDECREF(mod);
  
      ¦   /* XOptions is initialized after first some imports.
      ¦   ¦* So we can't have negative cache.
      ¦   ¦* Anyway, importlib.__find_and_load is much slower than
      ¦   ¦* _PyDict_GetItemId()
      ¦   ¦*/
      ¦   if (ximporttime == 0) {
      ¦   ¦   PyObject *xoptions = PySys_GetXOptions();
      ¦   ¦   if (xoptions) {
      ¦   ¦   ¦   PyObject *value = _PyDict_GetItemId(xoptions, &PyId_importtime);
      ¦   ¦   ¦   ximporttime = (value == Py_True);
      ¦   ¦   }
      ¦   ¦   if (ximporttime) {
      ¦   ¦   ¦   fputs("import time: self [us] | cumulative | imported package\n",
      ¦   ¦   ¦   ¦   ¦ stderr);
      ¦   ¦   }
      ¦   }
  
      ¦   if (ximporttime) {
      ¦   ¦   import_level++;
      ¦   ¦   t1 = _PyTime_GetMonotonicClock();
      ¦   ¦   accumulated = 0;
      ¦   }
      ¦   if (PyDTrace_IMPORT_FIND_LOAD_START_ENABLED())
      ¦   ¦   PyDTrace_IMPORT_FIND_LOAD_START(PyUnicode_AsUTF8(abs_name));
  
      ¦   mod = _PyObject_CallMethodIdObjArgs(interp->importlib,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   &PyId__find_and_load, abs_name,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   interp->import_func, NULL);
  
      ¦   if (PyDTrace_IMPORT_FIND_LOAD_DONE_ENABLED())
      ¦   ¦   PyDTrace_IMPORT_FIND_LOAD_DONE(PyUnicode_AsUTF8(abs_name),
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦  mod != NULL);
  
      ¦   if (ximporttime) {
      ¦   ¦   _PyTime_t cum = _PyTime_GetMonotonicClock() - t1;
  
      ¦   ¦   import_level--;
      ¦   ¦   fprintf(stderr, "import time: %9ld | %10ld | %*s%s\n",
      ¦   ¦   ¦   ¦   (long)_PyTime_AsMicroseconds(cum - accumulated, _PyTime_ROUND_CEILING),
      ¦   ¦   ¦   ¦   (long)_PyTime_AsMicroseconds(cum, _PyTime_ROUND_CEILING),
      ¦   ¦   ¦   ¦   import_level*2, "", PyUnicode_AsUTF8(abs_name));
  
      ¦   ¦   accumulated = accumulated_copy + cum;
      ¦   }
  
      ¦   if (mod == NULL) {
      ¦   ¦   goto error;
      ¦   }
      }
  
      has_from = 0;
      if (fromlist != NULL && fromlist != Py_None) {
      ¦   has_from = PyObject_IsTrue(fromlist);
      ¦   if (has_from < 0)
      ¦   ¦   goto error;
      }
      if (!has_from) {
      ¦   Py_ssize_t len = PyUnicode_GET_LENGTH(name);
      ¦   if (level == 0 || len > 0) {
      ¦   ¦   Py_ssize_t dot;
          ¦   dot = PyUnicode_FindChar(name, '.', 0, len, 1);
      ¦   ¦   if (dot == -2) {
      ¦   ¦   ¦   goto error;
      ¦   ¦   }
  
      ¦   ¦   if (dot == -1) {
      ¦   ¦   ¦   /* No dot in module name, simple exit */
      ¦   ¦   ¦   final_mod = mod;
      ¦   ¦   ¦   Py_INCREF(mod);
      ¦   ¦   ¦   goto error;
      ¦   ¦   }
  
      ¦   ¦   if (level == 0) {
      ¦   ¦   ¦   PyObject *front = PyUnicode_Substring(name, 0, dot);
      ¦   ¦   ¦   if (front == NULL) {
      ¦   ¦   ¦   ¦   goto error;
      ¦   ¦   ¦   }
  
      ¦   ¦   ¦   final_mod = PyImport_ImportModuleLevelObject(front, NULL, NULL, NULL, 0);
      ¦   ¦   ¦   Py_DECREF(front);
      ¦   ¦   }
      ¦   ¦   else {
      ¦   ¦   ¦   Py_ssize_t cut_off = len - dot;
      ¦   ¦   ¦   Py_ssize_t abs_name_len = PyUnicode_GET_LENGTH(abs_name);
      ¦   ¦   ¦   PyObject *to_return = PyUnicode_Substring(abs_name, 0,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   abs_name_len - cut_off);
      ¦   ¦   ¦   if (to_return == NULL) {
      ¦   ¦   ¦   ¦   goto error;
      ¦   ¦   ¦   }
  
      ¦   ¦   ¦   final_mod = PyImport_GetModule(to_return);
      ¦   ¦   ¦   Py_DECREF(to_return);
      ¦   ¦   ¦   if (final_mod == NULL) {
      ¦   ¦   ¦   ¦       ¦   ¦   ¦   ¦   PyErr_Format(PyExc_KeyError,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦"%R not in sys.modules as expected",
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦to_return);
      ¦   ¦   ¦   ¦   goto error;
      ¦   ¦   ¦   }
      ¦   ¦   }
      ¦   }
      ¦   else {
      ¦   ¦   final_mod = mod;
      ¦   ¦   Py_INCREF(mod);
      ¦   }
      }
      else {
      ¦   final_mod = _PyObject_CallMethodIdObjArgs(interp->importlib,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦ &PyId__handle_fromlist, mod,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦ fromlist, interp->import_func,
      ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦   ¦ NULL);
      }
  
    error:
      Py_XDECREF(abs_name);
      Py_XDECREF(mod);
      Py_XDECREF(package);
      if (final_mod == NULL)
      ¦   remove_importlib_frames();
      return final_mod;
  }

/* --- Core PyObject call functions ------------------------------- */
  
  PyObject *
  _PyObject_FastCallDict(PyObject *callable, PyObject **args, Py_ssize_t nargs,
      ¦   ¦   ¦   ¦   ¦  PyObject *kwargs)
  {
      /* _PyObject_FastCallDict() must not be called with an exception set,
      ¦  because it can clear it (directly or indirectly) and so the
      ¦  caller loses its exception */
      assert(!PyErr_Occurred());
  
      assert(callable != NULL);
      assert(nargs >= 0);
      assert(nargs == 0 || args != NULL);
      assert(kwargs == NULL || PyDict_Check(kwargs));
  
      if (PyFunction_Check(callable)) {
      ¦   return _PyFunction_FastCallDict(callable, args, nargs, kwargs);
      }
      else if (PyCFunction_Check(callable)) {
      ¦   return _PyCFunction_FastCallDict(callable, args, nargs, kwargs);
      }
      else {
      ¦   PyObject *argstuple, *result;
      ¦   ternaryfunc call;
  
      ¦   /* Slow-path: build a temporary tuple */
      ¦   call = callable->ob_type->tp_call;
      ¦   if (call == NULL) {
      ¦   ¦   PyErr_Format(PyExc_TypeError, "'%.200s' object is not callable",
      ¦   ¦   ¦   ¦   ¦   ¦callable->ob_type->tp_name);
      ¦   ¦   return NULL;
      ¦   }
  
      ¦   argstuple = _PyStack_AsTuple(args, nargs);
      ¦   if (argstuple == NULL) {
      ¦   ¦   return NULL;
      ¦   }

      ¦   if (Py_EnterRecursiveCall(" while calling a Python object")) {
      ¦   ¦   Py_DECREF(argstuple);
      ¦   ¦   return NULL;
      ¦   }
  
      ¦   result = (*call)(callable, argstuple, kwargs);
  
      ¦   Py_LeaveRecursiveCall();
      ¦   Py_DECREF(argstuple);
  
      ¦   result = _Py_CheckFunctionResult(callable, result, NULL);
      ¦   return result;
      }
  }

compile and eval

compile?

Signature: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1)
Docstring:
Compile source into a code object that can be executed by exec() or eval().

The source code may represent a Python module, statement or expression.
The filename will be used for run-time error messages.
The mode must be 'exec' to compile a module, 'single' to compile a
single (interactive) statement, or 'eval' to compile an expression.
The flags argument, if present, controls which future statements influence
the compilation of the code.
The dont_inherit argument, if true, stops the compilation inheriting
the effects of any future statements in effect in the code calling
compile; if absent or false these statements do influence the compilation,
in addition to any features explicitly specified.
Type:      builtin_function_or_method

bytecode

Python source code is compiled into bytecode, the internal representation of a Python program in the CPython interpreter. The bytecode is also cached in .pyc files so that executing the same file is faster the second time (recompilation from source to bytecode can be avoided). This “intermediate language” is said to run on a virtual machine that executes the machine code corresponding to each bytecode. Do note that bytecodes are not expected to work between different Python virtual machines, nor to be stable between Python releases.

A list of bytecode instructions can be found in the documentation for the dis module.

help(exec)

Help on built-in function exec in module builtins:

exec(source, globals=None, locals=None, /)
    Execute the given source in the context of globals and locals.
    
    The source may be a string representing one or more Python statements
    or a code object as returned by compile().
    The globals must be a dictionary and locals can be any mapping,
    defaulting to the current globals and locals.
    If only globals is given, locals defaults to it.

help(eval)

Help on built-in function eval in module builtins:

eval(source, globals=None, locals=None, /)
    Evaluate the given source in the context of globals and locals.
    
    The source may be a string representing a Python expression
    or a code object as returned by compile().
    The globals must be a dictionary and locals can be any mapping,
    defaulting to the current globals and locals.
    If only globals is given, locals defaults to it.

所以eval執行的內容必須是expression,而exec是都可以執行,但是回傳值固定為None。

code_str="""
def hello():
    print("hello world")
hello()
"""

exec('print(5)')           # prints 5.
# exec 'print 5'     if you use Python 2.x, nor the exec neither the print is a function there
exec('print(5)\nprint(6)')  # prints 5{newline}6.
exec('if True: print(6)')  # prints 6.
exec('5')

eval('5')              # x <- 5
eval('%d + 6' % x)     # x <- 11
eval('abs(%d)' % -100) # x <- 100
eval('x = 5')          # INVALID; assignment is not an expression.
eval('if 1: x = 4')    # INVALID; if is a statement, not an expression.

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-107-40888bad4179> in <module>()
      1 eval('5')              # x <- 5
----> 2 eval('%d + 6' % x)     # x <- 11
      3 eval('abs(%d)' % -100) # x <- 100
      4 eval('x = 5')          # INVALID; assignment is not an expression.
      5 eval('if 1: x = 4')    # INVALID; if is a statement, not an expression.


TypeError: %d format: a number is required, not generator

code_obj_exec=compile(code_str,'string','exec')
code_obj_exec
exec(code_obj_exec)

hello world

eval(compile('42', '<string>', 'exec'))  # code return None

eval(compile('42', '<string>', 'eval'))  # code returns 42

exec(compile('42', '<string>', 'eval'))  # code returns 42,but ignored by exec

code_obj_signle=compile(code_str,'string','single')
exec(code_obj_signle)

Traceback (most recent call last):


  File "/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)


  File "<ipython-input-112-e920440df200>", line 1, in <module>
    code_obj_signle=compile(code_str,'string','single')


  File "string", line 4
    hello()
        ^
SyntaxError: invalid syntax

而single 只接受oneline statment

code_obj_eval=compile(code_str,'string','eval')

Traceback (most recent call last):


  File "/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py", line 2862, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)


  File "<ipython-input-113-686a1487d9cc>", line 1, in <module>
    code_obj_eval=compile(code_str,'string','eval')


  File "string", line 2
    def hello():
      ^
SyntaxError: invalid syntax

exec('print("hello")')

hello

eval("print('hello')")

hello

exec("3+5")

eval("3+5")

Actually the statement "eval accepts only a single expression" applies only when a string (which contains Python source code) is passed to eval. Then it is internally compiled to bytecode using compile(source, '<string>', 'eval') This is where the difference really comes from.

If a code object (which contains Python bytecode) is passed to exec or eval, they behave identically, excepting for the fact that exec ignores the return value, still returning None always. So it is possible use eval to execute something that has statements, if you just compiled it into bytecode before instead of passing it as a string:

eval(compile('if 1: print("Hello")', '<string>', 'exec'))

Hello

It should work.

The longer answer, a.k.a the gory details

The exec function (which was a statement in Python 2) is used for executing a dynamically created statement or program.
The eval function does the same for a single expression, and returns the value of the expression.

exec and eval both accept the program/expression to be run either as a str, unicode or bytes object containing source code, or as a code object which contains Python bytecode.

If a str/unicode/bytes containing source code was passed to exec, it behaves equivalently to:

exec(compile(source, '<string>', 'exec'))

and eval similarly behaves equivalent to:

eval(compile(source, '<string>', 'eval'))

def my_func(arg):
    print("Called with %d" % arg)
    return arg * 2

exec('my_func(42)')

Called with 42

eval('my_func(42)')

Called with 42





84

深入code object

Code objects

Code objects represent byte-compiled executable Python code, or bytecode.
The difference between a code object and a function object is that the function object contains an explicit reference to the function’s globals (the module in which it was defined), while a code object contains no context; also the default argument values are stored in the function object, not in the code object (because they represent values calculated at run-time). Unlike function objects, code objects are immutable and contain no references (directly or indirectly) to mutable objects.

code	
    co_argcount	number of arguments (not including keyword only arguments, * or ** args)
 	co_code	string of raw compiled bytecode
 	co_cellvars	tuple of names of cell variables (referenced by containing scopes)
 	co_consts	tuple of constants used in the bytecode
 	co_filename	name of file in which this code object was created
 	co_firstlineno	number of first line in Python source code
 	co_flags	bitmap of CO_* flags, read more here
 	co_lnotab	encoded mapping of line numbers to bytecode indices
 	co_freevars	tuple of names of free variables (referenced via a function’s closure)
 	co_kwonlyargcount	number of keyword only arguments (not including ** arg)
 	co_name	name with which this code object was defined
 	co_names	tuple of names of local variables
 	co_nlocals	number of local variables
 	co_stacksize	virtual machine stack space required
 	co_varnames	tuple of names of arguments and local variables

可以參考python的inspect lib

code object implement

inside include/code.h

/* Bytecode object */
typedef struct {
    PyObject_HEAD int co_argcount; /* #arguments, except *args */
    int co_kwonlyargcount;         /* #keyword only arguments */
    int co_nlocals;                /* #local variables */
    int co_stacksize;              /* #entries needed for evaluation stack */
    int co_flags;                  /* CO_..., see below */
    int co_firstlineno;            /* first source line number */
    PyObject* co_code;             /* instruction opcodes */
    PyObject* co_consts;           /* list (constants used) */
    PyObject* co_names;            /* list of strings (names used) */
    PyObject* co_varnames;         /* tuple of strings (local variable names) */
    PyObject* co_freevars;         /* tuple of strings (free variable names) */
    PyObject* co_cellvars;         /* tuple of strings (cell variable names) */
    /* The rest aren't used in either hash or comparisons, except for co_name,
       used in both. This is done to preserve the name and line number
       for tracebacks and debuggers; otherwise, constant de-duplication
       would collapse identical functions/lambdas defined on different lines.
    */
    Py_ssize_t* co_cell2arg;  /* Maps cell vars which are arguments. */
    PyObject* co_filename;    /* unicode (where it was loaded from) */
    PyObject* co_name;        /* unicode (name, for reference) */
    PyObject* co_lnotab;      /* string (encoding addr<->lineno mapping) See
                                 Objects/lnotab_notes.txt for details. */
    void* co_zombieframe;     /* for optimization only (see frameobject.c) */
    PyObject* co_weakreflist; /* to support weakrefs to code objects */
    /* Scratch space for extra data relating to the code object.
       Type is a void* to keep the format private in codeobject.c to force
       people to go through the proper APIs. */
    void* co_extra;
} PyCodeObject;

more about code object

inside the code object

code_obj=compile(code_str,'string','exec')
dir(code_obj)

['__class__',
 '__delattr__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__le__',
 '__lt__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 'co_argcount',
 'co_cellvars',
 'co_code',
 'co_consts',
 'co_filename',
 'co_firstlineno',
 'co_flags',
 'co_freevars',
 'co_kwonlyargcount',
 'co_lnotab',
 'co_name',
 'co_names',
 'co_nlocals',
 'co_stacksize',
 'co_varnames']

get code object code

code_obj.co_code

b'd\x00\x00d\x01\x00\x84\x00\x00Z\x00\x00e\x00\x00\x83\x00\x00\x01d\x02\x00S'

[x for x in code_obj.co_code]

[100,
 0,
 0,
 100,
 1,
 0,
 132,
 0,
 0,
 90,
 0,
 0,
 101,
 0,
 0,
 131,
 0,
 0,
 1,
 100,
 2,
 0,
 83]

dis.show_code(code_obj)

Name:              <module>
Filename:          string
Argument count:    0
Kw-only arguments: 0
Number of locals:  0
Stack size:        2
Flags:             NOFREE
Constants:
   0: <code object hello at 0x7f01fc67f030, file "string", line 2>
   1: 'hello'
   2: None
Names:
   0: hello

dis.dis(code_obj)

  2           0 LOAD_CONST               0 (<code object hello at 0x7f01fc67f030, file "string", line 2>)
              3 LOAD_CONST               1 ('hello')
              6 MAKE_FUNCTION            0
              9 STORE_NAME               0 (hello)

  4          12 LOAD_NAME                0 (hello)
             15 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
             18 POP_TOP
             19 LOAD_CONST               2 (None)
             22 RETURN_VALUE

和上面的code object做對照

iterator=dis.get_instructions(code_str)
for it in iterator:
    print(it.opcode)

op=code_obj.co_code[0]
op

opcode.opname[op]

'LOAD_CONST'

可以發現opcode是python的一個module，他負責定義python interpreter會用到的instruction，所以我們才可以透過dis來disassembly python bytecode,而code object是bytecode的表現形式。

python中opcode的value和python virtual machine所定義的instruction相同(定義於opcode.h中，和dis.opmap的value互相對應)，因此可以透過實做interpreter(不管任何形式)產生的python bytecode來讓python virtual machine執行。

code_obj.co_consts

(<code object hello at 0x7f3768227d20, file "string", line 2>, 'hello', None)

code_obj.co_lnotab

b'\x0c\x03'

code_obj.co_freevars

()

code_obj.co_stacksize

A Detour into Code Disassembly

import dis
def hello():
    print("Hello world")

code_str="""
def print_hello(x,y):
    print("hello")
    return x+y
hello(10,11)
"""

code_obj=compile(code_str,'string','exec')
dis.dis(code_obj)

  2           0 LOAD_CONST               0 (<code object print_hello at 0x7f01fc6fde40, file "string", line 2>)
              3 LOAD_CONST               1 ('print_hello')
              6 MAKE_FUNCTION            0
              9 STORE_NAME               0 (print_hello)

  5          12 LOAD_NAME                1 (hello)
             15 LOAD_CONST               2 (10)
             18 LOAD_CONST               3 (11)
             21 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
             24 POP_TOP
             25 LOAD_CONST               4 (None)
             28 RETURN_VALUE

line number	bytecode offect	bytecode	bytecode argument
5	12	LOAD_NAME	0(hello)

bytecode=dis.Bytecode(hello)
bytecode

Bytecode(<function hello at 0x7f01fc74fd90>)

help(bytecode)

Help on Bytecode in module dis object:

class Bytecode(builtins.object)
 |  The bytecode operations of a piece of code
 |  
 |  Instantiate this with a function, method, string of code, or a code object
 |  (as returned by compile()).
 |  
 |  Iterating over this yields the bytecode operations as Instruction instances.
 |  
 |  Methods defined here:
 |  
 |  __init__(self, x, *, first_line=None, current_offset=None)
 |      Initialize self.  See help(type(self)) for accurate signature.
 |  
 |  __iter__(self)
 |  
 |  __repr__(self)
 |      Return repr(self).
 |  
 |  dis(self)
 |      Return a formatted view of the bytecode operations.
 |  
 |  info(self)
 |      Return formatted information about the code object.
 |  
 |  ----------------------------------------------------------------------
 |  Class methods defined here:
 |  
 |  from_traceback(tb) from builtins.type
 |      Construct a Bytecode from the given traceback
 |  
 |  ----------------------------------------------------------------------
 |  Data descriptors defined here:
 |  
 |  __dict__
 |      dictionary for instance variables (if defined)
 |  
 |  __weakref__
 |      list of weak references to the object (if defined)

bytecode.dis()

"  3           0 LOAD_GLOBAL              0 (print)\n              3 LOAD_CONST               1 ('Hello world')\n              6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)\n              9 POP_TOP\n             10 LOAD_CONST               0 (None)\n             13 RETURN_VALUE\n"

bytecode.info()

"Name:              hello\nFilename:          <ipython-input-70-9b74c1c13393>\nArgument count:    0\nKw-only arguments: 0\nNumber of locals:  0\nStack size:        2\nFlags:             OPTIMIZED, NEWLOCALS, NOFREE\nConstants:\n   0: None\n   1: 'Hello world'\nNames:\n   0: print"

for instr in bytecode:
    print(instr.opname)

LOAD_GLOBAL
LOAD_CONST
CALL_FUNCTION
POP_TOP
LOAD_CONST
RETURN_VALUE

dis.dis(hello)

  3           0 LOAD_GLOBAL              0 (print)
              3 LOAD_CONST               1 ('Hello world')
              6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
              9 POP_TOP
             10 LOAD_CONST               0 (None)
             13 RETURN_VALUE

import opcode
help(opcode)

Help on module opcode:

NAME
    opcode

MODULE REFERENCE
    https://docs.python.org/3.5/library/opcode
    
    The following documentation is automatically generated from the Python
    source files.  It may be incomplete, incorrect or include features that
    are considered implementation detail and may vary between Python
    implementations.  When in doubt, consult the module reference at the
    location listed above.

DESCRIPTION
    opcode module - potentially shared between dis and other modules which
    operate on bytecodes (e.g. peephole optimizers).

FUNCTIONS
    stack_effect(opcode, oparg=None, /)
        Compute the stack effect of the opcode.

DATA
    EXTENDED_ARG = 144
    HAVE_ARGUMENT = 90
    __all__ = ['cmp_op', 'hasconst', 'hasname', 'hasjrel', 'hasjabs', 'has...
    cmp_op = ('<', '<=', '==', '!=', '>', '>=', 'in', 'not in', 'is', 'is ...
    hascompare = [107]
    hasconst = [100]
    hasfree = [135, 136, 137, 138, 148]
    hasjabs = [111, 112, 113, 114, 115, 119]
    hasjrel = [93, 110, 120, 121, 122, 143, 154]
    haslocal = [124, 125, 126]
    hasname = [90, 91, 95, 96, 97, 98, 101, 106, 108, 109, 116]
    hasnargs = [131, 140, 141, 142]
    opmap = {'BEFORE_ASYNC_WITH': 52, 'BINARY_ADD': 23, 'BINARY_AND': 64, ...
    opname = ['<0>', 'POP_TOP', 'ROT_TWO', 'ROT_THREE', 'DUP_TOP', 'DUP_TO...

FILE
    /usr/lib/python3.5/opcode.py

dis.code_info(hello)

"Name:              hello\nFilename:          <ipython-input-70-9b74c1c13393>\nArgument count:    0\nKw-only arguments: 0\nNumber of locals:  0\nStack size:        2\nFlags:             OPTIMIZED, NEWLOCALS, NOFREE\nConstants:\n   0: None\n   1: 'Hello world'\nNames:\n   0: print"

dis.show_code(hello)

Name:              hello
Filename:          <ipython-input-70-9b74c1c13393>
Argument count:    0
Kw-only arguments: 0
Number of locals:  0
Stack size:        2
Flags:             OPTIMIZED, NEWLOCALS, NOFREE
Constants:
   0: None
   1: 'Hello world'
Names:
   0: print

help(dis.distb)

Help on function distb in module dis:

distb(tb=None, *, file=None)
    Disassemble a traceback (default: last traceback).

dis.disassemble(code_obj)

  2           0 LOAD_CONST               0 (<code object print_hello at 0x7f01fc6fde40, file "string", line 2>)
              3 LOAD_CONST               1 ('print_hello')
              6 MAKE_FUNCTION            0
              9 STORE_NAME               0 (print_hello)

  5          12 LOAD_NAME                1 (hello)
             15 LOAD_CONST               2 (10)
             18 LOAD_CONST               3 (11)
             21 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
             24 POP_TOP
             25 LOAD_CONST               4 (None)
             28 RETURN_VALUE

dis.disco(code_obj)

  2           0 LOAD_CONST               0 (<code object print_hello at 0x7f01fc6fde40, file "string", line 2>)
              3 LOAD_CONST               1 ('print_hello')
              6 MAKE_FUNCTION            0
              9 STORE_NAME               0 (print_hello)

  5          12 LOAD_NAME                1 (hello)
             15 LOAD_CONST               2 (10)
             18 LOAD_CONST               3 (11)
             21 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
             24 POP_TOP
             25 LOAD_CONST               4 (None)
             28 RETURN_VALUE

這兩個function的作用是一樣的

iterator=dis.get_instructions(code_str)
for it in iterator:
    print(it)

Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=<code object print_hello at 0x7f01fc6f5030, file "<disassembly>", line 2>, argrepr='<code object print_hello at 0x7f01fc6f5030, file "<disassembly>", line 2>', offset=0, starts_line=2, is_jump_target=False)
Instruction(opname='LOAD_CONST', opcode=100, arg=1, argval='print_hello', argrepr="'print_hello'", offset=3, starts_line=None, is_jump_target=False)
Instruction(opname='MAKE_FUNCTION', opcode=132, arg=0, argval=0, argrepr='', offset=6, starts_line=None, is_jump_target=False)
Instruction(opname='STORE_NAME', opcode=90, arg=0, argval='print_hello', argrepr='print_hello', offset=9, starts_line=None, is_jump_target=False)
Instruction(opname='LOAD_NAME', opcode=101, arg=1, argval='hello', argrepr='hello', offset=12, starts_line=5, is_jump_target=False)
Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=10, argrepr='10', offset=15, starts_line=None, is_jump_target=False)
Instruction(opname='LOAD_CONST', opcode=100, arg=3, argval=11, argrepr='11', offset=18, starts_line=None, is_jump_target=False)
Instruction(opname='CALL_FUNCTION', opcode=131, arg=2, argval=2, argrepr='2 positional, 0 keyword pair', offset=21, starts_line=None, is_jump_target=False)
Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=24, starts_line=None, is_jump_target=False)
Instruction(opname='LOAD_CONST', opcode=100, arg=4, argval=None, argrepr='None', offset=25, starts_line=None, is_jump_target=False)
Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=28, starts_line=None, is_jump_target=False)

opname - human readable name for operation
opcode - numeric code for operation
arg - numeric argument to operation (if any), otherwise None
argval - resolved arg value (if known), otherwise same as arg
argrepr - human readable description of operation argument
offset - start index of operation within bytecode sequence
starts_line - line started by this opcode (if any), otherwise None
is_jump_target - True if other code jumps to here, otherwise False

x=dis.findlinestarts(code_obj)
next(x)

(0, 2)

next(x)

(12, 5)

help(dis.stack_effect)

Help on built-in function stack_effect in module _opcode:

stack_effect(opcode, oparg=None, /)
    Compute the stack effect of the opcode.

dis.stack_effect(1) # POP_TOP

-1

help(dis.findlabels)

Help on function findlabels in module dis:

findlabels(code)
    Detect all offsets in a byte code which are jump targets.
    
    Return the list of offsets.

def f(x):
    if x == 0:
        return 1
    elif x==1:
        return 2

dis.disco(f.__code__)

  2           0 LOAD_FAST                0 (x)
              3 LOAD_CONST               1 (0)
              6 COMPARE_OP               2 (==)
              9 POP_JUMP_IF_FALSE       16

  3          12 LOAD_CONST               2 (1)
             15 RETURN_VALUE

  4     >>   16 LOAD_FAST                0 (x)
             19 LOAD_CONST               2 (1)
             22 COMPARE_OP               2 (==)
             25 POP_JUMP_IF_FALSE       32

  5          28 LOAD_CONST               3 (2)
             31 RETURN_VALUE
        >>   32 LOAD_CONST               0 (None)
             35 RETURN_VALUE

dis.findlabels(f.__code__.co_code)

[16, 32]

dis.opname

['<0>',
 'POP_TOP',
 'ROT_TWO',
 'ROT_THREE',
 'DUP_TOP',
 'DUP_TOP_TWO',
 '<6>',
 '<7>',
 '<8>',
 'NOP',
 'UNARY_POSITIVE',
 'UNARY_NEGATIVE',
 'UNARY_NOT',
 '<13>',
 '<14>',
 'UNARY_INVERT',
 'BINARY_MATRIX_MULTIPLY',
 'INPLACE_MATRIX_MULTIPLY',
 '<18>',
 'BINARY_POWER',
 'BINARY_MULTIPLY',
 '<21>',
 'BINARY_MODULO',
 'BINARY_ADD',
 'BINARY_SUBTRACT',
 'BINARY_SUBSCR',
 'BINARY_FLOOR_DIVIDE',
 'BINARY_TRUE_DIVIDE',
 'INPLACE_FLOOR_DIVIDE',
 'INPLACE_TRUE_DIVIDE',
 '<30>',
 '<31>',
 '<32>',
 '<33>',
 '<34>',
 '<35>',
 '<36>',
 '<37>',
 '<38>',
 '<39>',
 '<40>',
 '<41>',
 '<42>',
 '<43>',
 '<44>',
 '<45>',
 '<46>',
 '<47>',
 '<48>',
 '<49>',
 'GET_AITER',
 'GET_ANEXT',
 'BEFORE_ASYNC_WITH',
 '<53>',
 '<54>',
 'INPLACE_ADD',
 'INPLACE_SUBTRACT',
 'INPLACE_MULTIPLY',
 '<58>',
 'INPLACE_MODULO',
 'STORE_SUBSCR',
 'DELETE_SUBSCR',
 'BINARY_LSHIFT',
 'BINARY_RSHIFT',
 'BINARY_AND',
 'BINARY_XOR',
 'BINARY_OR',
 'INPLACE_POWER',
 'GET_ITER',
 'GET_YIELD_FROM_ITER',
 'PRINT_EXPR',
 'LOAD_BUILD_CLASS',
 'YIELD_FROM',
 'GET_AWAITABLE',
 '<74>',
 'INPLACE_LSHIFT',
 'INPLACE_RSHIFT',
 'INPLACE_AND',
 'INPLACE_XOR',
 'INPLACE_OR',
 'BREAK_LOOP',
 'WITH_CLEANUP_START',
 'WITH_CLEANUP_FINISH',
 'RETURN_VALUE',
 'IMPORT_STAR',
 '<85>',
 'YIELD_VALUE',
 'POP_BLOCK',
 'END_FINALLY',
 'POP_EXCEPT',
 'STORE_NAME',
 'DELETE_NAME',
 'UNPACK_SEQUENCE',
 'FOR_ITER',
 'UNPACK_EX',
 'STORE_ATTR',
 'DELETE_ATTR',
 'STORE_GLOBAL',
 'DELETE_GLOBAL',
 '<99>',
 'LOAD_CONST',
 'LOAD_NAME',
 'BUILD_TUPLE',
 'BUILD_LIST',
 'BUILD_SET',
 'BUILD_MAP',
 'LOAD_ATTR',
 'COMPARE_OP',
 'IMPORT_NAME',
 'IMPORT_FROM',
 'JUMP_FORWARD',
 'JUMP_IF_FALSE_OR_POP',
 'JUMP_IF_TRUE_OR_POP',
 'JUMP_ABSOLUTE',
 'POP_JUMP_IF_FALSE',
 'POP_JUMP_IF_TRUE',
 'LOAD_GLOBAL',
 '<117>',
 '<118>',
 'CONTINUE_LOOP',
 'SETUP_LOOP',
 'SETUP_EXCEPT',
 'SETUP_FINALLY',
 '<123>',
 'LOAD_FAST',
 'STORE_FAST',
 'DELETE_FAST',
 '<127>',
 '<128>',
 '<129>',
 'RAISE_VARARGS',
 'CALL_FUNCTION',
 'MAKE_FUNCTION',
 'BUILD_SLICE',
 'MAKE_CLOSURE',
 'LOAD_CLOSURE',
 'LOAD_DEREF',
 'STORE_DEREF',
 'DELETE_DEREF',
 '<139>',
 'CALL_FUNCTION_VAR',
 'CALL_FUNCTION_KW',
 'CALL_FUNCTION_VAR_KW',
 'SETUP_WITH',
 'EXTENDED_ARG',
 'LIST_APPEND',
 'SET_ADD',
 'MAP_ADD',
 'LOAD_CLASSDEREF',
 'BUILD_LIST_UNPACK',
 'BUILD_MAP_UNPACK',
 'BUILD_MAP_UNPACK_WITH_CALL',
 'BUILD_TUPLE_UNPACK',
 'BUILD_SET_UNPACK',
 'SETUP_ASYNC_WITH',
 '<155>',
 '<156>',
 '<157>',
 '<158>',
 '<159>',
 '<160>',
 '<161>',
 '<162>',
 '<163>',
 '<164>',
 '<165>',
 '<166>',
 '<167>',
 '<168>',
 '<169>',
 '<170>',
 '<171>',
 '<172>',
 '<173>',
 '<174>',
 '<175>',
 '<176>',
 '<177>',
 '<178>',
 '<179>',
 '<180>',
 '<181>',
 '<182>',
 '<183>',
 '<184>',
 '<185>',
 '<186>',
 '<187>',
 '<188>',
 '<189>',
 '<190>',
 '<191>',
 '<192>',
 '<193>',
 '<194>',
 '<195>',
 '<196>',
 '<197>',
 '<198>',
 '<199>',
 '<200>',
 '<201>',
 '<202>',
 '<203>',
 '<204>',
 '<205>',
 '<206>',
 '<207>',
 '<208>',
 '<209>',
 '<210>',
 '<211>',
 '<212>',
 '<213>',
 '<214>',
 '<215>',
 '<216>',
 '<217>',
 '<218>',
 '<219>',
 '<220>',
 '<221>',
 '<222>',
 '<223>',
 '<224>',
 '<225>',
 '<226>',
 '<227>',
 '<228>',
 '<229>',
 '<230>',
 '<231>',
 '<232>',
 '<233>',
 '<234>',
 '<235>',
 '<236>',
 '<237>',
 '<238>',
 '<239>',
 '<240>',
 '<241>',
 '<242>',
 '<243>',
 '<244>',
 '<245>',
 '<246>',
 '<247>',
 '<248>',
 '<249>',
 '<250>',
 '<251>',
 '<252>',
 '<253>',
 '<254>',
 '<255>']

sorted(dis.opmap.items(),key=lambda x:x[1])

[('POP_TOP', 1),
 ('ROT_TWO', 2),
 ('ROT_THREE', 3),
 ('DUP_TOP', 4),
 ('DUP_TOP_TWO', 5),
 ('NOP', 9),
 ('UNARY_POSITIVE', 10),
 ('UNARY_NEGATIVE', 11),
 ('UNARY_NOT', 12),
 ('UNARY_INVERT', 15),
 ('BINARY_MATRIX_MULTIPLY', 16),
 ('INPLACE_MATRIX_MULTIPLY', 17),
 ('BINARY_POWER', 19),
 ('BINARY_MULTIPLY', 20),
 ('BINARY_MODULO', 22),
 ('BINARY_ADD', 23),
 ('BINARY_SUBTRACT', 24),
 ('BINARY_SUBSCR', 25),
 ('BINARY_FLOOR_DIVIDE', 26),
 ('BINARY_TRUE_DIVIDE', 27),
 ('INPLACE_FLOOR_DIVIDE', 28),
 ('INPLACE_TRUE_DIVIDE', 29),
 ('GET_AITER', 50),
 ('GET_ANEXT', 51),
 ('BEFORE_ASYNC_WITH', 52),
 ('INPLACE_ADD', 55),
 ('INPLACE_SUBTRACT', 56),
 ('INPLACE_MULTIPLY', 57),
 ('INPLACE_MODULO', 59),
 ('STORE_SUBSCR', 60),
 ('DELETE_SUBSCR', 61),
 ('BINARY_LSHIFT', 62),
 ('BINARY_RSHIFT', 63),
 ('BINARY_AND', 64),
 ('BINARY_XOR', 65),
 ('BINARY_OR', 66),
 ('INPLACE_POWER', 67),
 ('GET_ITER', 68),
 ('GET_YIELD_FROM_ITER', 69),
 ('PRINT_EXPR', 70),
 ('LOAD_BUILD_CLASS', 71),
 ('YIELD_FROM', 72),
 ('GET_AWAITABLE', 73),
 ('INPLACE_LSHIFT', 75),
 ('INPLACE_RSHIFT', 76),
 ('INPLACE_AND', 77),
 ('INPLACE_XOR', 78),
 ('INPLACE_OR', 79),
 ('BREAK_LOOP', 80),
 ('WITH_CLEANUP_START', 81),
 ('WITH_CLEANUP_FINISH', 82),
 ('RETURN_VALUE', 83),
 ('IMPORT_STAR', 84),
 ('YIELD_VALUE', 86),
 ('POP_BLOCK', 87),
 ('END_FINALLY', 88),
 ('POP_EXCEPT', 89),
 ('STORE_NAME', 90),
 ('DELETE_NAME', 91),
 ('UNPACK_SEQUENCE', 92),
 ('FOR_ITER', 93),
 ('UNPACK_EX', 94),
 ('STORE_ATTR', 95),
 ('DELETE_ATTR', 96),
 ('STORE_GLOBAL', 97),
 ('DELETE_GLOBAL', 98),
 ('LOAD_CONST', 100),
 ('LOAD_NAME', 101),
 ('BUILD_TUPLE', 102),
 ('BUILD_LIST', 103),
 ('BUILD_SET', 104),
 ('BUILD_MAP', 105),
 ('LOAD_ATTR', 106),
 ('COMPARE_OP', 107),
 ('IMPORT_NAME', 108),
 ('IMPORT_FROM', 109),
 ('JUMP_FORWARD', 110),
 ('JUMP_IF_FALSE_OR_POP', 111),
 ('JUMP_IF_TRUE_OR_POP', 112),
 ('JUMP_ABSOLUTE', 113),
 ('POP_JUMP_IF_FALSE', 114),
 ('POP_JUMP_IF_TRUE', 115),
 ('LOAD_GLOBAL', 116),
 ('CONTINUE_LOOP', 119),
 ('SETUP_LOOP', 120),
 ('SETUP_EXCEPT', 121),
 ('SETUP_FINALLY', 122),
 ('LOAD_FAST', 124),
 ('STORE_FAST', 125),
 ('DELETE_FAST', 126),
 ('RAISE_VARARGS', 130),
 ('CALL_FUNCTION', 131),
 ('MAKE_FUNCTION', 132),
 ('BUILD_SLICE', 133),
 ('MAKE_CLOSURE', 134),
 ('LOAD_CLOSURE', 135),
 ('LOAD_DEREF', 136),
 ('STORE_DEREF', 137),
 ('DELETE_DEREF', 138),
 ('CALL_FUNCTION_VAR', 140),
 ('CALL_FUNCTION_KW', 141),
 ('CALL_FUNCTION_VAR_KW', 142),
 ('SETUP_WITH', 143),
 ('EXTENDED_ARG', 144),
 ('LIST_APPEND', 145),
 ('SET_ADD', 146),
 ('MAP_ADD', 147),
 ('LOAD_CLASSDEREF', 148),
 ('BUILD_LIST_UNPACK', 149),
 ('BUILD_MAP_UNPACK', 150),
 ('BUILD_MAP_UNPACK_WITH_CALL', 151),
 ('BUILD_TUPLE_UNPACK', 152),
 ('BUILD_SET_UNPACK', 153),
 ('SETUP_ASYNC_WITH', 154)]

len(dis.opmap.items()) # numbers of the python instructions

而這些bytecode被定義在opcode module中

opcode??

dis.cmp_op

('<',
 '<=',
 '==',
 '!=',
 '>',
 '>=',
 'in',
 'not in',
 'is',
 'is not',
 'exception match',
 'BAD')

dis.hasconst

[100]

dis.hasfree

[135, 136, 137, 138, 148]

dis.hasname

[90, 91, 95, 96, 97, 98, 101, 106, 108, 109, 116]

dis.hasjrel

[93, 110, 120, 121, 122, 143, 154]

dis.hasjabs

[111, 112, 113, 114, 115, 119]

dis.haslocal

[124, 125, 126]

dis.hascompare

[107]

Ref

Python dis module

https://docs.python.org/3.7/library/dis.html?module-dis

stack overflow 對於比較exec和eval的回答

https://stackoverflow.com/questions/2220699/whats-the-difference-between-eval-exec-and-compile-in-python

Python virtual machine implementation

Python/ceval.c

PyObject* _Py_HOT_FUNCTION
_PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag)

python interpreter main的進入點

Py_Main(int arg,wchar_t argv**)

python interpreter initiali

_Py_InitializeCore

PyFrameObject
Include/frameobject.h

typedef struct _frame {
    PyObject_VAR_HEAD
    struct _frame *f_back;      /* previous frame, or NULL */
    PyCodeObject *f_code;       /* code segment */
    PyObject *f_builtins;       /* builtin symbol table (PyDictObject) */
    PyObject *f_globals;        /* global symbol table (PyDictObject) */
    PyObject *f_locals;         /* local symbol table (any mapping) */
    PyObject **f_valuestack;    /* points after the last local */
    /* Next free slot in f_valuestack.  Frame creation sets to f_valuestack.
       Frame evaluation usually NULLs it, but a frame that yields sets it
       to the current stack top. */
    PyObject **f_stacktop;
    PyObject *f_trace;          /* Trace function */

    /* In a generator, we need to be able to swap between the exception
       state inside the generator and the exception state of the calling
       frame (which shouldn't be impacted when the generator "yields"
       from an except handler).
       These three fields exist exactly for that, and are unused for
       non-generator frames. See the save_exc_state and swap_exc_state
       functions in ceval.c for details of their use. */
    PyObject *f_exc_type, *f_exc_value, *f_exc_traceback;
    /* Borrowed reference to a generator, or NULL */
    PyObject *f_gen;

    int f_lasti;                /* Last instruction if called */
    /* Call PyFrame_GetLineNumber() instead of reading this field
       directly.  As of 2.3 f_lineno is only valid when tracing is
       active (i.e. when f_trace is set).  At other times we use
       PyCode_Addr2Line to calculate the line from the current
       bytecode index. */
    int f_lineno;               /* Current line number */
    int f_iblock;               /* index in f_blockstack */
    char f_executing;           /* whether the frame is still executing */
    PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */
    PyObject *f_localsplus[1];  /* locals+stack, dynamically sized */
} PyFrameObject;

python object


/* Define pointers to support a doubly-linked list of all live heap objects. */
#define _PyObject_HEAD_EXTRA            \
    struct _object *_ob_next;           \
    struct _object *_ob_prev;

#define _PyObject_EXTRA_INIT 0, 0,

#else
#define _PyObject_HEAD_EXTRA
#define _PyObject_EXTRA_INIT
#endif

/* PyObject_HEAD defines the initial segment of every PyObject. */
#define PyObject_HEAD                   PyObject ob_base;

#define PyObject_HEAD_INIT(type)        \
    { _PyObject_EXTRA_INIT              \
    1, type },

#define PyVarObject_HEAD_INIT(type, size)       \
    { PyObject_HEAD_INIT(type) size },

/* PyObject_VAR_HEAD defines the initial segment of all variable-size
 * container objects.  These end with a declaration of an array with 1
 * element, but enough space is malloc'ed so that the array actually
 * has room for ob_size elements.  Note that ob_size is an element count,
 * not necessarily a byte count.
 */
#define PyObject_VAR_HEAD      PyVarObject ob_base;
#define Py_INVALID_SIZE (Py_ssize_t)-1

/* Nothing is actually declared to be a PyObject, but every pointer to
 * a Python object can be cast to a PyObject*.  This is inheritance built
 * by hand.  Similarly every pointer to a variable-size Python object can,
 * in addition, be cast to PyVarObject*.
 */
typedef struct _object {
    _PyObject_HEAD_EXTRA
    Py_ssize_t ob_refcnt;
    struct _typeobject *ob_type;
} PyObject;

typedef struct {
    PyObject ob_base;
    Py_ssize_t ob_size; /* Number of items in variable part */
} PyVarObject;

#define Py_REFCNT(ob)           (((PyObject*)(ob))->ob_refcnt)
#define Py_TYPE(ob)             (((PyObject*)(ob))->ob_type)
#define Py_SIZE(ob)             (((PyVarObject*)(ob))->ob_size)


typedef void (*freefunc)(void *);
typedef void (*destructor)(PyObject *);
#ifndef Py_LIMITED_API
/* We can't provide a full compile-time check that limited-API
   users won't implement tp_print. However, not defining printfunc
   and making tp_print of a different function pointer type
   should at least cause a warning in most cases. */
typedef int (*printfunc)(PyObject *, FILE *, int);
#endif
typedef PyObject *(*getattrfunc)(PyObject *, char *);
typedef PyObject *(*getattrofunc)(PyObject *, PyObject *);
typedef int (*setattrfunc)(PyObject *, char *, PyObject *);
typedef int (*setattrofunc)(PyObject *, PyObject *, PyObject *);
typedef PyObject *(*reprfunc)(PyObject *);
typedef Py_hash_t (*hashfunc)(PyObject *);
typedef PyObject *(*richcmpfunc) (PyObject *, PyObject *, int);
typedef PyObject *(*getiterfunc) (PyObject *);
typedef PyObject *(*iternextfunc) (PyObject *);
typedef PyObject *(*descrgetfunc) (PyObject *, PyObject *, PyObject *);
typedef int (*descrsetfunc) (PyObject *, PyObject *, PyObject *);
typedef int (*initproc)(PyObject *, PyObject *, PyObject *);
typedef PyObject *(*newfunc)(struct _typeobject *, PyObject *, PyObject *);
typedef PyObject *(*allocfunc)(struct _typeobject *, Py_ssize_t);

#ifdef Py_LIMITED_API
typedef struct _typeobject PyTypeObject; /* opaque */
#else
typedef struct _typeobject {
    PyObject_VAR_HEAD
    const char *tp_name; /* For printing, in format "<module>.<name>" */
    Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */

    /* Methods to implement standard operations */

    destructor tp_dealloc;
    printfunc tp_print;
    getattrfunc tp_getattr;
    setattrfunc tp_setattr;
    PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2)
                                    or tp_reserved (Python 3) */
    reprfunc tp_repr;

    /* Method suites for standard classes */

    PyNumberMethods *tp_as_number;
    PySequenceMethods *tp_as_sequence;
    PyMappingMethods *tp_as_mapping;

    /* More standard operations (here for binary compatibility) */

    hashfunc tp_hash;
    ternaryfunc tp_call;
    reprfunc tp_str;
    getattrofunc tp_getattro;
    setattrofunc tp_setattro;

    /* Functions to access object as input/output buffer */
    PyBufferProcs *tp_as_buffer;

    /* Flags to define presence of optional/expanded features */
    unsigned long tp_flags;

    const char *tp_doc; /* Documentation string */

    /* Assigned meaning in release 2.0 */
    /* call function for all accessible objects */
    traverseproc tp_traverse;

    /* delete references to contained objects */
    inquiry tp_clear;

    /* Assigned meaning in release 2.1 */
    /* rich comparisons */
    richcmpfunc tp_richcompare;

    /* weak reference enabler */
    Py_ssize_t tp_weaklistoffset;

    /* Iterators */
    getiterfunc tp_iter;
    iternextfunc tp_iternext;

    /* Attribute descriptor and subclassing stuff */
    struct PyMethodDef *tp_methods;
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
    struct _typeobject *tp_base;
    PyObject *tp_dict;
    descrgetfunc tp_descr_get;
    descrsetfunc tp_descr_set;
    Py_ssize_t tp_dictoffset;
    initproc tp_init;
    allocfunc tp_alloc;
    newfunc tp_new;
    freefunc tp_free; /* Low-level free-memory routine */
    inquiry tp_is_gc; /* For PyObject_IS_GC */
    PyObject *tp_bases;
    PyObject *tp_mro; /* method resolution order */
    PyObject *tp_cache;
    PyObject *tp_subclasses;
    PyObject *tp_weaklist;
    destructor tp_del;

    /* Type attribute cache version tag. Added in version 2.6 */
    unsigned int tp_version_tag;

    destructor tp_finalize;

#ifdef COUNT_ALLOCS
    /* these must be last and never explicitly initialized */
    Py_ssize_t tp_allocs;
    Py_ssize_t tp_frees;
    Py_ssize_t tp_maxalloc;
    struct _typeobject *tp_prev;
    struct _typeobject *tp_next;
#endif
} PyTypeObject;
#endif

Design of CPython’s Compiler

2017-08-25 12:30:51

In CPython, the compilation from source code to bytecode involves several steps:

Parse source code into a parse tree (Parser/pgen.c)
Transform parse tree into an Abstract Syntax Tree (Python/ast.c)
Transform AST into a Control Flow Graph (Python/compile.c)
Emit bytecode based on the Control Flow Graph (Python/compile.c)

Important Files

Parser/
- Python.asdl
  ASDL syntax file
- asdl.py
  Parser for ASDL definition files. Reads in an ASDL description and parses it into an AST that describes it.
- asdl_c.py
  “Generate C code from an ASDL description.” Generates Python/Python-ast.c and Include/Python-ast.h.
Python/
- Python-ast.c
  Creates C structs corresponding to the ASDL types. Also contains code for marshalling AST nodes (core ASDL types have marshalling code in asdl.c). “File automatically generated by Parser/asdl_c.py”. This file must be committed separately after every grammar change is committed since the version value is set to the latest grammar change revision number.
- asdl.c
  Contains code to handle the ASDL sequence type. Also has code to handle marshalling the core ASDL types, such as number and identifier. Used by Python-ast.c for marshalling AST nodes.
- ast.c
  Converts Python’s parse tree into the abstract syntax tree.
- ceval.c
  Executes byte code (aka, eval loop).
- compile.c
  Emits bytecode based on the AST.
- symtable.c
  Generates a symbol table from AST.
- pyarena.c
  Implementation of the arena memory manager.
- import.c
  Home of the magic number (named MAGIC) for bytecode versioning
Include/
- Python-ast.h
  Contains the actual definitions of the C structs as generated by Python/Python-ast.c. “Automatically generated by Parser/asdl_c.py”.
- asdl.h
  Header for the corresponding Python/ast.c.
- ast.h
  Declares PyAST_FromNode() external (from Python/ast.c).
- code.h
  Header file for Objects/codeobject.c; contains definition of PyCodeObject.
- symtable.h
  Header for Python/symtable.c. struct symtable and PySTEntryObject are defined here.
- pyarena.h
  Header file for the corresponding Python/pyarena.c.
- opcode.h
  Master list of bytecode; if this file is modified you must modify several other files accordingly (see “Introducing New Bytecode”)
Objects/
- codeobject.c
  Contains PyCodeObject-related code (originally in Python/compile.c).
Lib/
- opcode.py
  One of the files that must be modified if Include/opcode.h is.

Ref

https://docs.python.org/devguide/compiler.html
http://blog.csdn.net/zhsenl/article/category/2307109/1
https://leanpub.com/insidethepythonvirtualmachine/read#leanpub-auto-the-block-stack
python pyc