error accessing external object property 怎么解决????急!!急!!急!!急!!急!!急!!急!!急!!急!!

dengmingmeng 2009-08-14 02:54:51
string ls_bz,ls_bb,ls_cc
long i
i = lenw(ls_bz)
ls_bb = left(ls_bz,500)
ls_cc = right(ls_bz,lenw(ls_bz)-500)
if i<= 1000 then
dw_1.object.text_bz.text = ls_bb
if messagebox("",) then
dw_1.print()
end if
else
dw_2.object.text_bz.text = ls_bb
//每次运行至此就报错!!!
如果我换成dw_2.object.text_bz.text = '大幅度开水'
这样的字符串就没有错,或者是dw_2.object.text_bz.text = ls_bz
也不会报错,就是用了变量名ls_bb或者是ls_cc就报错!
但是我在上面dw_1.object.text_bz.text = ls_bb
这里用这个变量又不会报错!!!!!
为什么???????????????

dw_3.object.text_bz.text = string(ls_cc)
//同样的问题
if messagebox("",) then
dw_2.print()
end if
if messagebox("",) then
dw_3.print()
end if
end if
...全文
9337 14 打赏 收藏 转发到动态 举报
AI 作业
写回复
用AI写文章
14 条回复
切换为时间正序
请发表友善的回复…
发表回复
meiven 2009-08-15
  • 打赏
  • 举报
回复
[Quote=引用 10 楼 xys_777 的回复:]
ls_bz里面有些字符有问题,可能会与pb的关键字冲突,这种情况我见多了
[/Quote]
[Quote=引用 4 楼 jlwei888 的回复:]
ls_bb = left(ls_bz,500)
ls_cc = right(ls_bz,lenw(ls_bz)-500)
看看这两个值是什么吧!好像截的有问题!
[/Quote]

真可能“ls_bz里面有些字符有问题”,输出ls_bb、ls_cc看看是什么吧!
zhshaoya 2009-08-15
  • 打赏
  • 举报
回复
判断一下 ls_bb 值是不是 null
if isnull(ls_bb) or trim(ls_bb) = "" then ls_bb=""
dengmingmeng 2009-08-14
  • 打赏
  • 举报
回复
[Quote=引用 9 楼 dev1978 的回复:]
ls_bb,ls_zz为空了
[/Quote]

这个我没有看懂,我不知道哪里置空了?
dengmingmeng 2009-08-14
  • 打赏
  • 举报
回复
[Quote=引用 8 楼 sdhylj 的回复:]
C/C++ codestring ls_bz,ls_bb,ls_cclong i
i= lenw(ls_bz)
ls_bb= left(ls_bz,500)
ls_cc= right(ls_bz,lenw(ls_bz)-500)
这是完整的代码吗?
ls_bz为何没有初始化?
[/Quote]
ls_bz 有初始化的,我没有写上去!
永生天地 2009-08-14
  • 打赏
  • 举报
回复
ls_bz里面有些字符有问题,可能会与pb的关键字冲突,这种情况我见多了
Powertion 2009-08-14
  • 打赏
  • 举报
回复
ls_bb,ls_zz为空了
青锋-SS 2009-08-14
  • 打赏
  • 举报
回复
string ls_bz,ls_bb,ls_cc
long i
i = lenw(ls_bz)
ls_bb = left(ls_bz,500)
ls_cc = right(ls_bz,lenw(ls_bz)-500)

这是完整的代码吗?
ls_bz为何没有初始化?
dengmingmeng 2009-08-14
  • 打赏
  • 举报
回复
[Quote=引用 3 楼 fuyonghao 的回复:]
error accessing external object property

dw_2和dw_3中的text_bz是文本还是列名。。。。。是否有错误。。。是否存在。。。。
[/Quote]

这个没有问题,我输出过!!!!!!
dengmingmeng 2009-08-14
  • 打赏
  • 举报
回复
[Quote=引用 3 楼 fuyonghao 的回复:]
error accessing external object property

dw_2和dw_3中的text_bz是文本还是列名。。。。。是否有错误。。。是否存在。。。。
[/Quote]
dw_2和dw_3中的text_bz是文本!!!!!
青锋-SS 2009-08-14
  • 打赏
  • 举报
回复
看标题上的提示应该是dw里面的对象有问题,看楼主的描述又不像.
jlwei888 2009-08-14
  • 打赏
  • 举报
回复
ls_bb = left(ls_bz,500)
ls_cc = right(ls_bz,lenw(ls_bz)-500)
看看这两个值是什么吧!好像截的有问题!
FuYongHao 2009-08-14
  • 打赏
  • 举报
回复
error accessing external object property

dw_2和dw_3中的text_bz是文本还是列名。。。。。是否有错误。。。是否存在。。。。
FuYongHao 2009-08-14
  • 打赏
  • 举报
回复
什么错误信息???
青锋-SS 2009-08-14
  • 打赏
  • 举报
回复
好像没有问题,不过对前几名有些不解,ls_bz的值从何而来?
greenplum-db-6.2.1-rhel7-x86_64.rpm Pivotal Greenplum 6.2 Release Notes This document contains pertinent release information about Pivotal Greenplum Database 6.2 releases. For previous versions of the release notes for Greenplum Database, go to Pivotal Greenplum Database Documentation. For information about Greenplum Database end of life, see Pivotal Greenplum Database end of life policy. Pivotal Greenplum 6 software is available for download from the Pivotal Greenplum page on Pivotal Network. Pivotal Greenplum 6 is based on the open source Greenplum Database project code. Important: Pivotal Support does not provide support for open source versions of Greenplum Database. Only Pivotal Greenplum Database is supported by Pivotal Support. Release 6.2.1 Release Date: 2019-12-12 Pivotal Greenplum 6.2.1 is a minor release that includes new features and resolves several issues. New Features Greenplum Database 6.2.1 includes these new features: Greenplum Database supports materialized views. Materialized views are similar to views. A materialized view enables you to save a frequently used or complex query, then access the query results in a SELECT statement as if they were a table. Materialized views persist the query results in a table-like form. Materialized view data cannot be directly updated. To refresh the materialized view data, use the REFRESH MATERIALIZED VIEW command. See Creating and Managing Materialized Views. Note: Known Issues and Limitations describes a limitation of materialized view support in Greenplum 6.2.1. The gpinitsystem utility supports the --ignore-warnings option. The option controls the value returned by gpinitsystem when warnings or an error occurs. If you specify this option, gpinitsystem returns 0 if warnings occurred during system initialization, and returns a non-zero value if a fatal error occurs. If this option is not specified, gpinitsystem returns 1 if initialization completes with warnings, and returns value of 2 or greater if a fatal error occurs. PXF version 5.10.0 is included, which introduces several new and changed features and bug fixes. See PXF Version 5.10.0 below. PXF Version 5.10.0 PXF 5.10.0 includes the following new and changed features: PXF has improved its performance when reading a large number of files from HDFS or an object store. PXF bundles newer tomcat and jackson libraries. The PXF JDBC Connector now supports pushdown of OR and NOT logical filter operators when specified in a JDBC named query or in an external table query filter condition. PXF supports writing Avro-format data to Hadoop and object stores. Refer to Reading and Writing HDFS Avro Data for more information about this feature. PXF is now certified with Hadoop 2.x and 3.1.x and Hive Server 2.x and 3.1, and bundles new and upgraded Hadoop libraries to support these versions. PXF supports Kerberos authentication to Hive Server 2.x and 3.1.x. PXF supports per-server user impersonation configuration. PXF supports concurrent access to multiple Kerberized Hadoop clusters. In previous releases of Greenplum Database, PXF supported accessing a single Hadoop cluster secured with Kerberos, and this Hadoop cluster must have been configured as the default PXF server. PXF introduces a new template file, pxf-site.xml, to specify the Kerberos and impersonation property settings for a Hadoop or JDBC server configuration. Refer to About Kerberos and User Impersonation Configuration (pxf-site.xml) for more information about this file. PXF now supports connecting to Hadoop with a configurable Hadoop user identity. PXF previously supported only proxy access to Hadoop via the gpadmin Greenplum user. PXF version 5.10.0 deprecates the following configuration properties. Note: These property settings continue to work. The PXF_USER_IMPERSONATION, PXF_PRINCIPAL, and PXF_KEYTAB settings in the pxf-env.sh file. You can use the pxf-site.xml file to configure Kerberos and impersonation settings for your new Hadoop server configurations. The pxf.impersonation.jdbc property setting in the jdbc-site.xml file. You can use the pxf.service.user.impersonation property to configure user impersonation for a new JDBC server configuration. Note: If you have previously configured a PXF JDBC server to access Kerberos-secured Hive, you must upgrade the server definition. See Upgrading PXF in Greenplum 6.x for more information. Changed Features Greenplum Database 6.2.1 includes these changed features: Greenplum Stream Server version 1.3.1 is included in the Greenplum distribution. Resolved Issues Pivotal Greenplum 6.2.1 is a minor release that resolves these issues: 29454 - gpstart During Greenplum Database start up, the gpstart utility did not report when a segment instance failed to start. The utility always displayed 0 skipped segment starts. This issue has been resolved. gpstart output was also enhanced to provide additional warnings and summary information about the number of skipped segments. For example: [WARNING]:-********
Python参考手册,官方正式版参考手册,chm版。以下摘取部分内容:Navigation index modules | next | Python » 3.6.5 Documentation » Python Documentation contents What’s New in Python What’s New In Python 3.6 Summary – Release highlights New Features PEP 498: Formatted string literals PEP 526: Syntax for variable annotations PEP 515: Underscores in Numeric Literals PEP 525: Asynchronous Generators PEP 530: Asynchronous Comprehensions PEP 487: Simpler customization of class creation PEP 487: Descriptor Protocol Enhancements PEP 519: Adding a file system path protocol PEP 495: Local Time Disambiguation PEP 529: Change Windows filesystem encoding to UTF-8 PEP 528: Change Windows console encoding to UTF-8 PEP 520: Preserving Class Attribute Definition Order PEP 468: Preserving Keyword Argument Order New dict implementation PEP 523: Adding a frame evaluation API to CPython PYTHONMALLOC environment variable DTrace and SystemTap probing support Other Language Changes New Modules secrets Improved Modules array ast asyncio binascii cmath collections concurrent.futures contextlib datetime decimal distutils email encodings enum faulthandler fileinput hashlib http.client idlelib and IDLE importlib inspect json logging math multiprocessing os pathlib pdb pickle pickletools pydoc random re readline rlcompleter shlex site sqlite3 socket socketserver ssl statistics struct subprocess sys telnetlib time timeit tkinter traceback tracemalloc typing unicodedata unittest.mock urllib.request urllib.robotparser venv warnings winreg winsound xmlrpc.client zipfile zlib Optimizations Build and C API Changes Other Improvements Deprecated New Keywords Deprecated Python behavior Deprecated Python modules, functions and methods asynchat asyncore dbm distutils grp importlib os re ssl tkinter venv Deprecated functions and types of the C API Deprecated Build Options Removed API and Feature Removals Porting to Python 3.6 Changes in ‘python’ Command Behavior Changes in the Python API Changes in the C API CPython bytecode changes Notable changes in Python 3.6.2 New make regen-all build target Removal of make touch build target Notable changes in Python 3.6.5 What’s New In Python 3.5 Summary – Release highlights New Features PEP 492 - Coroutines with async and await syntax PEP 465 - A dedicated infix operator for matrix multiplication PEP 448 - Additional Unpacking Generalizations PEP 461 - percent formatting support for bytes and bytearray PEP 484 - Type Hints PEP 471 - os.scandir() function – a better and faster directory iterator PEP 475: Retry system calls failing with EINTR PEP 479: Change StopIteration handling inside generators PEP 485: A function for testing approximate equality PEP 486: Make the Python Launcher aware of virtual environments PEP 488: Elimination of PYO files PEP 489: Multi-phase extension module initialization Other Language Changes New Modules typing zipapp Improved Modules argparse asyncio bz2 cgi cmath code collections collections.abc compileall concurrent.futures configparser contextlib csv curses dbm difflib distutils doctest email enum faulthandler functools glob gzip heapq http http.client idlelib and IDLE imaplib imghdr importlib inspect io ipaddress json linecache locale logging lzma math multiprocessing operator os pathlib pickle poplib re readline selectors shutil signal smtpd smtplib sndhdr socket ssl Memory BIO Support Application-Layer Protocol Negotiation Support Other Changes sqlite3 subprocess sys sysconfig tarfile threading time timeit tkinter traceback types unicodedata unittest unittest.mock urllib wsgiref xmlrpc xml.sax zipfile Other module-level changes Optimizations Build and C API Changes Deprecated New Keywords Deprecated Python Behavior Unsupported Operating Systems Deprecated Python modules, functions and methods Removed API and Feature Removals Porting to Python 3.5 Changes in Python behavior Changes in the Python API Changes in the C API What’s New In Python 3.4 Summary – Release Highlights New Features PEP 453: Explicit Bootstrapping of PIP in Python Installations Bootstrapping pip By Default Documentation Changes PEP 446: Newly Created File Descriptors Are Non-Inheritable Improvements to Codec Handling PEP 451: A ModuleSpec Type for the Import System Other Language Changes New Modules asyncio ensurepip enum pathlib selectors statistics tracemalloc Improved Modules abc aifc argparse audioop base64 collections colorsys contextlib dbm dis doctest email filecmp functools gc glob hashlib hmac html http idlelib and IDLE importlib inspect ipaddress logging marshal mmap multiprocessing operator os pdb pickle plistlib poplib pprint pty pydoc re resource select shelve shutil smtpd smtplib socket sqlite3 ssl stat struct subprocess sunau sys tarfile textwrap threading traceback types urllib unittest venv wave weakref xml.etree zipfile CPython Implementation Changes PEP 445: Customization of CPython Memory Allocators PEP 442: Safe Object Finalization PEP 456: Secure and Interchangeable Hash Algorithm PEP 436: Argument Clinic Other Build and C API Changes Other Improvements Significant Optimizations Deprecated Deprecations in the Python API Deprecated Features Removed Operating Systems No Longer Supported API and Feature Removals Code Cleanups Porting to Python 3.4 Changes in ‘python’ Command Behavior Changes in the Python API Changes in the C API Changed in 3.4.3 PEP 476: Enabling certificate verification by default for stdlib http clients What’s New In Python 3.3 Summary – Release highlights PEP 405: Virtual Environments PEP 420: Implicit Namespace Packages PEP 3118: New memoryview implementation and buffer protocol documentation Features API changes PEP 393: Flexible String Representation Functionality Performance and resource usage PEP 397: Python Launcher for Windows PEP 3151: Reworking the OS and IO exception hierarchy PEP 380: Syntax for Delegating to a Subgenerator PEP 409: Suppressing exception context PEP 414: Explicit Unicode literals PEP 3155: Qualified name for classes and functions PEP 412: Key-Sharing Dictionary PEP 362: Function Signature Object PEP 421: Adding sys.implementation SimpleNamespace Using importlib as the Implementation of Import New APIs Visible Changes Other Language Changes A Finer-Grained Import Lock Builtin functions and types New Modules faulthandler ipaddress lzma Improved Modules abc array base64 binascii bz2 codecs collections contextlib crypt curses datetime decimal Features API changes email Policy Framework Provisional Policy with New Header API Other API Changes ftplib functools gc hmac http html imaplib inspect io itertools logging math mmap multiprocessing nntplib os pdb pickle pydoc re sched select shlex shutil signal smtpd smtplib socket socketserver sqlite3 ssl stat struct subprocess sys tarfile tempfile textwrap threading time types unittest urllib webbrowser xml.etree.ElementTree zlib Optimizations Build and C API Changes Deprecated Unsupported Operating Systems Deprecated Python modules, functions and methods Deprecated functions and types of the C API Deprecated features Porting to Python 3.3 Porting Python code Porting C code Building C extensions Command Line Switch Changes What’s New In Python 3.2 PEP 384: Defining a Stable ABI PEP 389: Argparse Command Line Parsing Module PEP 391: Dictionary Based Configuration for Logging PEP 3148: The concurrent.futures module PEP 3147: PYC Repository Directories PEP 3149: ABI Version Tagged .so Files PEP 3333: Python Web Server Gateway Interface v1.0.1 Other Language Changes New, Improved, and Deprecated Modules email elementtree functools itertools collections threading datetime and time math abc io reprlib logging csv contextlib decimal and fractions ftp popen select gzip and zipfile tarfile hashlib ast os shutil sqlite3 html socket ssl nntp certificates imaplib http.client unittest random poplib asyncore tempfile inspect pydoc dis dbm ctypes site sysconfig pdb configparser urllib.parse mailbox turtledemo Multi-threading Optimizations Unicode Codecs Documentation IDLE Code Repository Build and C API Changes Porting to Python 3.2 What’s New In Python 3.1 PEP 372: Ordered Dictionaries PEP 378: Format Specifier for Thousands Separator Other Language Changes New, Improved, and Deprecated Modules Optimizations IDLE Build and C API Changes Porting to Python 3.1 What’s New In Python 3.0 Common Stumbling Blocks Print Is A Function Views And Iterators Instead Of Lists Ordering Comparisons Integers Text Vs. Data Instead Of Unicode Vs. 8-bit Overview Of Syntax Changes New Syntax Changed Syntax Removed Syntax Changes Already Present In Python 2.6 Library Changes PEP 3101: A New Approach To String Formatting Changes To Exceptions Miscellaneous Other Changes Operators And Special Methods Builtins Build and C API Changes Performance Porting To Python 3.0 What’s New in Python 2.7 The Future for Python 2.x Changes to the Handling of Deprecation Warnings Python 3.1 Features PEP 372: Adding an Ordered Dictionary to collections PEP 378: Format Specifier for Thousands Separator PEP 389: The argparse Module for Parsing Command Lines PEP 391: Dictionary-Based Configuration For Logging PEP 3106: Dictionary Views PEP 3137: The memoryview Object Other Language Changes Interpreter Changes Optimizations New and Improved Modules New module: importlib New module: sysconfig ttk: Themed Widgets for Tk Updated module: unittest Updated module: ElementTree 1.3 Build and C API Changes Capsules Port-Specific Changes: Windows Port-Specific Changes: Mac OS X Port-Specific Changes: FreeBSD Other Changes and Fixes Porting to Python 2.7 New Features Added to Python 2.7 Maintenance Releases PEP 434: IDLE Enhancement Exception for All Branches PEP 466: Network Security Enhancements for Python 2.7 Acknowledgements What’s New in Python 2.6 Python 3.0 Changes to the Development Process New Issue Tracker: Roundup New Documentation Format: reStructuredText Using Sphinx PEP 343: The ‘with’ statement Writing Context Managers The contextlib module PEP 366: Explicit Relative Imports From a Main Module PEP 370: Per-user site-packages Directory PEP 371: The multiprocessing Package PEP 3101: Advanced String Formatting PEP 3105: print As a Function PEP 3110: Exception-Handling Changes PEP 3112: Byte Literals PEP 3116: New I/O Library PEP 3118: Revised Buffer Protocol PEP 3119: Abstract Base Classes PEP 3127: Integer Literal Support and Syntax PEP 3129: Class Decorators PEP 3141: A Type Hierarchy for Numbers The fractions Module Other Language Changes Optimizations Interpreter Changes New and Improved Modules The ast module The future_builtins module The json module: JavaScript Object Notation The plistlib module: A Property-List Parser ctypes Enhancements Improved SSL Support Deprecations and Removals Build and C API Changes Port-Specific Changes: Windows Port-Specific Changes: Mac OS X Port-Specific Changes: IRIX Porting to Python 2.6 Acknowledgements What’s New in Python 2.5 PEP 308: Conditional Expressions PEP 309: Partial Function Application PEP 314: Metadata for Python Software Packages v1.1 PEP 328: Absolute and Relative Imports PEP 338: Executing Modules as Scripts PEP 341: Unified try/except/finally PEP 342: New Generator Features PEP 343: The ‘with’ statement Writing Context Managers The contextlib module PEP 352: Exceptions as New-Style Classes PEP 353: Using ssize_t as the index type PEP 357: The ‘__index__’ method Other Language Changes Interactive Interpreter Changes Optimizations New, Improved, and Removed Modules The ctypes package The ElementTree package The hashlib package The sqlite3 package The wsgiref package Build and C API Changes Port-Specific Changes Porting to Python 2.5 Acknowledgements What’s New in Python 2.4 PEP 218: Built-In Set Objects PEP 237: Unifying Long Integers and Integers PEP 289: Generator Expressions PEP 292: Simpler String Substitutions PEP 318: Decorators for Functions and Methods PEP 322: Reverse Iteration PEP 324: New subprocess Module PEP 327: Decimal Data Type Why is Decimal needed? The Decimal type The Context type PEP 328: Multi-line Imports PEP 331: Locale-Independent Float/String Conversions Other Language Changes Optimizations New, Improved, and Deprecated Modules cookielib doctest Build and C API Changes Port-Specific Changes Porting to Python 2.4 Acknowledgements What’s New in Python 2.3 PEP 218: A Standard Set Datatype PEP 255: Simple Generators PEP 263: Source Code Encodings PEP 273: Importing Modules from ZIP Archives PEP 277: Unicode file name support for Windows NT PEP 278: Universal Newline Support PEP 279: enumerate() PEP 282: The logging Package PEP 285: A Boolean Type PEP 293: Codec Error Handling Callbacks PEP 301: Package Index and Metadata for Distutils PEP 302: New Import Hooks PEP 305: Comma-separated Files PEP 307: Pickle Enhancements Extended Slices Other Language Changes String Changes Optimizations New, Improved, and Deprecated Modules Date/Time Type The optparse Module Pymalloc: A Specialized Object Allocator Build and C API Changes Port-Specific Changes Other Changes and Fixes Porting to Python 2.3 Acknowledgements What’s New in Python 2.2 Introduction PEPs 252 and 253: Type and Class Changes Old and New Classes Descriptors Multiple Inheritance: The Diamond Rule Attribute Access Related Links PEP 234: Iterators PEP 255: Simple Generators PEP 237: Unifying Long Integers and Integers PEP 238: Changing the Division Operator Unicode Changes PEP 227: Nested Scopes New and Improved Modules Interpreter Changes and Fixes Other Changes and Fixes Acknowledgements What’s New in Python 2.1 Introduction PEP 227: Nested Scopes PEP 236: __future__ Directives PEP 207: Rich Comparisons PEP 230: Warning Framework PEP 229: New Build System PEP 205: Weak References PEP 232: Function Attributes PEP 235: Importing Modules on Case-Insensitive Platforms PEP 217: Interactive Display Hook PEP 208: New Coercion Model PEP 241: Metadata in Python Packages New and Improved Modules Other Changes and Fixes Acknowledgements What’s New in Python 2.0 Introduction What About Python 1.6? New Development Process Unicode List Comprehensions Augmented Assignment String Methods Garbage Collection of Cycles Other Core Changes Minor Language Changes Changes to Built-in Functions Porting to 2.0 Extending/Embedding Changes Distutils: Making Modules Easy to Install XML Modules SAX2 Support DOM Support Relationship to PyXML Module changes New modules IDLE Improvements Deleted and Deprecated Modules Acknowledgements Changelog Python 3.6.5 final? Tests Build Python 3.6.5 release candidate 1? Security Core and Builtins Library Documentation Tests Build Windows macOS IDLE Tools/Demos C API Python 3.6.4 final? Python 3.6.4 release candidate 1? Core and Builtins Library Documentation Tests Build Windows macOS IDLE Tools/Demos C API Python 3.6.3 final? Library Build Python 3.6.3 release candidate 1? Security Core and Builtins Library Documentation Tests Build Windows IDLE Tools/Demos Python 3.6.2 final? Python 3.6.2 release candidate 2? Security Python 3.6.2 release candidate 1? Core and Builtins Library Security Library IDLE C API Build Documentation Tools/Demos Tests Windows Python 3.6.1 final? Core and Builtins Build Python 3.6.1 release candidate 1? Core and Builtins Library IDLE Windows C API Documentation Tests Build Python 3.6.0 final? Python 3.6.0 release candidate 2? Core and Builtins Tools/Demos Windows Build Python 3.6.0 release candidate 1? Core and Builtins Library C API Documentation Tools/Demos Python 3.6.0 beta 4? Core and Builtins Library Documentation Tests Build Python 3.6.0 beta 3? Core and Builtins Library Windows Build Tests Python 3.6.0 beta 2? Core and Builtins Library Windows C API Build Tests Python 3.6.0 beta 1? Core and Builtins Library IDLE C API Tests Build Tools/Demos Windows Python 3.6.0 alpha 4? Core and Builtins Library IDLE Tests Windows Build Python 3.6.0 alpha 3? Core and Builtins Library Security Library Security Library IDLE C API Build Tools/Demos Documentation Tests Python 3.6.0 alpha 2? Core and Builtins Library Security Library Security Library IDLE Documentation Tests Windows Build Windows C API Tools/Demos Python 3.6.0 alpha 1? Core and Builtins Library Security Library Security Library Security Library IDLE Documentation Tests Build Windows Tools/Demos C API Python 3.5.3 final? Python 3.5.3 release candidate 1? Core and Builtins Library Security Library Security Library IDLE C API Documentation Tests Tools/Demos Windows Build Python 3.5.2 final? Core and Builtins Tests IDLE Python 3.5.2 release candidate 1? Core and Builtins Security Library Security Library Security Library Security Library Security Library IDLE Documentation Tests Build Windows Tools/Demos Windows Python 3.5.1 final? Core and Builtins Windows Python 3.5.1 release candidate 1? Core and Builtins Library IDLE Documentation Tests Build Windows Tools/Demos Python 3.5.0 final? Build Python 3.5.0 release candidate 4? Library Build Python 3.5.0 release candidate 3? Core and Builtins Library Python 3.5.0 release candidate 2? Core and Builtins Library Python 3.5.0 release candidate 1? Core and Builtins Library IDLE Documentation Tests Python 3.5.0 beta 4? Core and Builtins Library Build Python 3.5.0 beta 3? Core and Builtins Library Tests Documentation Build Python 3.5.0 beta 2? Core and Builtins Library Python 3.5.0 beta 1? Core and Builtins Library IDLE Tests Documentation Tools/Demos Python 3.5.0 alpha 4? Core and Builtins Library Build Tests Tools/Demos C API Python 3.5.0 alpha 3? Core and Builtins Library Build Tests Tools/Demos Python 3.5.0 alpha 2? Core and Builtins Library Build C API Windows Python 3.5.0 alpha 1? Core and Builtins Library IDLE Build C API Documentation Tests Tools/Demos Windows The Python Tutorial 1. Whetting Your Appetite 2. Using the Python Interpreter 2.1. Invoking the Interpreter 2.1.1. Argument Passing 2.1.2. Interactive Mode 2.2. The Interpreter and Its Environment 2.2.1. Source Code Encoding 3. An Informal Introduction to Python 3.1. Using Python as a Calculator 3.1.1. Numbers 3.1.2. Strings 3.1.3. Lists 3.2. First Steps Towards Programming 4. More Control Flow Tools 4.1. if Statements 4.2. for Statements 4.3. The range() Function 4.4. break and continue Statements, and else Clauses on Loops 4.5. pass Statements 4.6. Defining Functions 4.7. More on Defining Functions 4.7.1. Default Argument Values 4.7.2. Keyword Arguments 4.7.3. Arbitrary Argument Lists 4.7.4. Unpacking Argument Lists 4.7.5. Lambda Expressions 4.7.6. Documentation Strings 4.7.7. Function Annotations 4.8. Intermezzo: Coding Style 5. Data Structures 5.1. More on Lists 5.1.1. Using Lists as Stacks 5.1.2. Using Lists as Queues 5.1.3. List Comprehensions 5.1.4. Nested List Comprehensions 5.2. The del statement 5.3. Tuples and Sequences 5.4. Sets 5.5. Dictionaries 5.6. Looping Techniques 5.7. More on Conditions 5.8. Comparing Sequences and Other Types 6. Modules 6.1. More on Modules 6.1.1. Executing modules as scripts 6.1.2. The Module Search Path 6.1.3. “Compiled” Python files 6.2. Standard Modules 6.3. The dir() Function 6.4. Packages 6.4.1. Importing * From a Package 6.4.2. Intra-package References 6.4.3. Packages in Multiple Directories 7. Input and Output 7.1. Fancier Output Formatting 7.1.1. Old string formatting 7.2. Reading and Writing Files 7.2.1. Methods of File Objects 7.2.2. Saving structured data with json 8. Errors and Exceptions 8.1. Syntax Errors 8.2. Exceptions 8.3. Handling Exceptions 8.4. Raising Exceptions 8.5. User-defined Exceptions 8.6. Defining Clean-up Actions 8.7. Predefined Clean-up Actions 9. Classes 9.1. A Word About Names and Objects 9.2. Python Scopes and Namespaces 9.2.1. Scopes and Namespaces Example 9.3. A First Look at Classes 9.3.1. Class Definition Syntax 9.3.2. Class Objects 9.3.3. Instance Objects 9.3.4. Method Objects 9.3.5. Class and Instance Variables 9.4. Random Remarks 9.5. Inheritance 9.5.1. Multiple Inheritance 9.6. Private Variables 9.7. Odds and Ends 9.8. Iterators 9.9. Generators 9.10. Generator Expressions 10. Brief Tour of the Standard Library 10.1. Operating System Interface 10.2. File Wildcards 10.3. Command Line Arguments 10.4. Error Output Redirection and Program Termination 10.5. String Pattern Matching 10.6. Mathematics 10.7. Internet Access 10.8. Dates and Times 10.9. Data Compression 10.10. Performance Measurement 10.11. Quality Control 10.12. Batteries Included 11. Brief Tour of the Standard Library — Part II 11.1. Output Formatting 11.2. Templating 11.3. Working with Binary Data Record Layouts 11.4. Multi-threading 11.5. Logging 11.6. Weak References 11.7. Tools for Working with Lists 11.8. Decimal Floating Point Arithmetic 12. Virtual Environments and Packages 12.1. Introduction 12.2. Creating Virtual Environments 12.3. Managing Packages with pip 13. What Now? 14. Interactive Input Editing and History Substitution 14.1. Tab Completion and History Editing 14.2. Alternatives to the Interactive Interpreter 15. Floating Point Arithmetic: Issues and Limitations 15.1. Representation Error 16. Appendix 16.1. Interactive Mode 16.1.1. Error Handling 16.1.2. Executable Python Scripts 16.1.3. The Interactive Startup File 16.1.4. The Customization Modules Python Setup and Usage 1. Command line and environment 1.1. Command line 1.1.1. Interface options 1.1.2. Generic options 1.1.3. Miscellaneous options 1.1.4. Options you shouldn’t use 1.2. Environment variables 1.2.1. Debug-mode variables 2. Using Python on Unix platforms 2.1. Getting and installing the latest version of Python 2.1.1. On Linux 2.1.2. On FreeBSD and OpenBSD 2.1.3. On OpenSolaris 2.2. Building Python 2.3. Python-related paths and files 2.4. Miscellaneous 2.5. Editors and IDEs 3. Using Python on Windows 3.1. Installing Python 3.1.1. Supported Versions 3.1.2. Installation Steps 3.1.3. Removing the MAX_PATH Limitation 3.1.4. Installing Without UI 3.1.5. Installing Without Downloading 3.1.6. Modifying an install 3.1.7. Other Platforms 3.2. Alternative bundles 3.3. Configuring Python 3.3.1. Excursus: Setting environment variables 3.3.2. Finding the Python executable 3.4. Python Launcher for Windows 3.4.1. Getting started 3.4.1.1. From the command-line 3.4.1.2. Virtual environments 3.4.1.3. From a script 3.4.1.4. From file associations 3.4.2. Shebang Lines 3.4.3. Arguments in shebang lines 3.4.4. Customization 3.4.4.1. Customization via INI files 3.4.4.2. Customizing default Python versions 3.4.5. Diagnostics 3.5. Finding modules 3.6. Additional modules 3.6.1. PyWin32 3.6.2. cx_Freeze 3.6.3. WConio 3.7. Compiling Python on Windows 3.8. Embedded Distribution 3.8.1. Python Application 3.8.2. Embedding Python 3.9. Other resources 4. Using Python on a Macintosh 4.1. Getting and Installing MacPython 4.1.1. How to run a Python script 4.1.2. Running scripts with a GUI 4.1.3. Configuration 4.2. The IDE 4.3. Installing Additional Python Packages 4.4. GUI Programming on the Mac 4.5. Distributing Python Applications on the Mac 4.6. Other Resources The Python Language Reference 1. Introduction 1.1. Alternate Implementations 1.2. Notation 2. Lexical analysis 2.1. Line structure 2.1.1. Logical lines 2.1.2. Physical lines 2.1.3. Comments 2.1.4. Encoding declarations 2.1.5. Explicit line joining 2.1.6. Implicit line joining 2.1.7. Blank lines 2.1.8. Indentation 2.1.9. Whitespace between tokens 2.2. Other tokens 2.3. Identifiers and keywords 2.3.1. Keywords 2.3.2. Reserved classes of identifiers 2.4. Literals 2.4.1. String and Bytes literals 2.4.2. String literal concatenation 2.4.3. Formatted string literals 2.4.4. Numeric literals 2.4.5. Integer literals 2.4.6. Floating point literals 2.4.7. Imaginary literals 2.5. Operators 2.6. Delimiters 3. Data model 3.1. Objects, values and types 3.2. The standard type hierarchy 3.3. Special method names 3.3.1. Basic customization 3.3.2. Customizing attribute access 3.3.2.1. Customizing module attribute access 3.3.2.2. Implementing Descriptors 3.3.2.3. Invoking Descriptors 3.3.2.4. __slots__ 3.3.2.4.1. Notes on using __slots__ 3.3.3. Customizing class creation 3.3.3.1. Metaclasses 3.3.3.2. Determining the appropriate metaclass 3.3.3.3. Preparing the class namespace 3.3.3.4. Executing the class body 3.3.3.5. Creating the class object 3.3.3.6. Metaclass example 3.3.4. Customizing instance and subclass checks 3.3.5. Emulating callable objects 3.3.6. Emulating container types 3.3.7. Emulating numeric types 3.3.8. With Statement Context Managers 3.3.9. Special method lookup 3.4. Coroutines 3.4.1. Awaitable Objects 3.4.2. Coroutine Objects 3.4.3. Asynchronous Iterators 3.4.4. Asynchronous Context Managers 4. Execution model 4.1. Structure of a program 4.2. Naming and binding 4.2.1. Binding of names 4.2.2. Resolution of names 4.2.3. Builtins and restricted execution 4.2.4. Interaction with dynamic features 4.3. Exceptions 5. The import system 5.1. importlib 5.2. Packages 5.2.1. Regular packages 5.2.2. Namespace packages 5.3. Searching 5.3.1. The module cache 5.3.2. Finders and loaders 5.3.3. Import hooks 5.3.4. The meta path 5.4. Loading 5.4.1. Loaders 5.4.2. Submodules 5.4.3. Module spec 5.4.4. Import-related module attributes 5.4.5. module.__path__ 5.4.6. Module reprs 5.5. The Path Based Finder 5.5.1. Path entry finders 5.5.2. Path entry finder protocol 5.6. Replacing the standard import system 5.7. Special considerations for __main__ 5.7.1. __main__.__spec__ 5.8. Open issues 5.9. References 6. Expressions 6.1. Arithmetic conversions 6.2. Atoms 6.2.1. Identifiers (Names) 6.2.2. Literals 6.2.3. Parenthesized forms 6.2.4. Displays for lists, sets and dictionaries 6.2.5. List displays 6.2.6. Set displays 6.2.7. Dictionary displays 6.2.8. Generator expressions 6.2.9. Yield expressions 6.2.9.1. Generator-iterator methods 6.2.9.2. Examples 6.2.9.3. Asynchronous generator functions 6.2.9.4. Asynchronous generator-iterator methods 6.3. Primaries 6.3.1. Attribute references 6.3.2. Subscriptions 6.3.3. Slicings 6.3.4. Calls 6.4. Await expression 6.5. The power operator 6.6. Unary arithmetic and bitwise operations 6.7. Binary arithmetic operations 6.8. Shifting operations 6.9. Binary bitwise operations 6.10. Comparisons 6.10.1. Value comparisons 6.10.2. Membership test operations 6.10.3. Identity comparisons 6.11. Boolean operations 6.12. Conditional expressions 6.13. Lambdas 6.14. Expression lists 6.15. Evaluation order 6.16. Operator precedence 7. Simple statements 7.1. Expression statements 7.2. Assignment statements 7.2.1. Augmented assignment statements 7.2.2. Annotated assignment statements 7.3. The assert statement 7.4. The pass statement 7.5. The del statement 7.6. The return statement 7.7. The yield statement 7.8. The raise statement 7.9. The break statement 7.10. The continue statement 7.11. The import statement 7.11.1. Future statements 7.12. The global statement 7.13. The nonlocal statement 8. Compound statements 8.1. The if statement 8.2. The while statement 8.3. The for statement 8.4. The try statement 8.5. The with statement 8.6. Function definitions 8.7. Class definitions 8.8. Coroutines 8.8.1. Coroutine function definition 8.8.2. The async for statement 8.8.3. The async with statement 9. Top-level components 9.1. Complete Python programs 9.2. File input 9.3. Interactive input 9.4. Expression input 10. Full Grammar specification The Python Standard Library 1. Introduction 2. Built-in Functions 3. Built-in Constants 3.1. Constants added by the site module 4. Built-in Types 4.1. Truth Value Testing 4.2. Boolean Operations — and, or, not 4.3. Comparisons 4.4. Numeric Types — int, float, complex 4.4.1. Bitwise Operations on Integer Types 4.4.2. Additional Methods on Integer Types 4.4.3. Additional Methods on Float 4.4.4. Hashing of numeric types 4.5. Iterator Types 4.5.1. Generator Types 4.6. Sequence Types — list, tuple, range 4.6.1. Common Sequence Operations 4.6.2. Immutable Sequence Types 4.6.3. Mutable Sequence Types 4.6.4. Lists 4.6.5. Tuples 4.6.6. Ranges 4.7. Text Sequence Type — str 4.7.1. String Methods 4.7.2. printf-style String Formatting 4.8. Binary Sequence Types — bytes, bytearray, memoryview 4.8.1. Bytes Objects 4.8.2. Bytearray Objects 4.8.3. Bytes and Bytearray Operations 4.8.4. printf-style Bytes Formatting 4.8.5. Memory Views 4.9. Set Types — set, frozenset 4.10. Mapping Types — dict 4.10.1. Dictionary view objects 4.11. Context Manager Types 4.12. Other Built-in Types 4.12.1. Modules 4.12.2. Classes and Class Instances 4.12.3. Functions 4.12.4. Methods 4.12.5. Code Objects 4.12.6. Type Objects 4.12.7. The Null Object 4.12.8. The Ellipsis Object 4.12.9. The NotImplemented Object 4.12.10. Boolean Values 4.12.11. Internal Objects 4.13. Special Attributes 5. Built-in Exceptions 5.1. Base classes 5.2. Concrete exceptions 5.2.1. OS exceptions 5.3. Warnings 5.4. Exception hierarchy 6. Text Processing Services 6.1. string — Common string operations 6.1.1. String constants 6.1.2. Custom String Formatting 6.1.3. Format String Syntax 6.1.3.1. Format Specification Mini-Language 6.1.3.2. Format examples 6.1.4. Template strings 6.1.5. Helper functions 6.2. re — Regular expression operations 6.2.1. Regular Expression Syntax 6.2.2. Module Contents 6.2.3. Regular Expression Objects 6.2.4. Match Objects 6.2.5. Regular Expression Examples 6.2.5.1. Checking for a Pair 6.2.5.2. Simulating scanf() 6.2.5.3. search() vs. match() 6.2.5.4. Making a Phonebook 6.2.5.5. Text Munging 6.2.5.6. Finding all Adverbs 6.2.5.7. Finding all Adverbs and their Positions 6.2.5.8. Raw String Notation 6.2.5.9. Writing a Tokenizer 6.3. difflib — Helpers for computing deltas 6.3.1. SequenceMatcher Objects 6.3.2. SequenceMatcher Examples 6.3.3. Differ Objects 6.3.4. Differ Example 6.3.5. A command-line interface to difflib 6.4. textwrap — Text wrapping and filling 6.5. unicodedata — Unicode Database 6.6. stringprep — Internet String Preparation 6.7. readline — GNU readline interface 6.7.1. Init file 6.7.2. Line buffer 6.7.3. History file 6.7.4. History list 6.7.5. Startup hooks 6.7.6. Completion 6.7.7. Example 6.8. rlcompleter — Completion function for GNU readline 6.8.1. Completer Objects 7. Binary Data Services 7.1. struct — Interpret bytes as packed binary data 7.1.1. Functions and Exceptions 7.1.2. Format Strings 7.1.2.1. Byte Order, Size, and Alignment 7.1.2.2. Format Characters 7.1.2.3. Examples 7.1.3. Classes 7.2. codecs — Codec registry and base classes 7.2.1. Codec Base Classes 7.2.1.1. Error Handlers 7.2.1.2. Stateless Encoding and Decoding 7.2.1.3. Incremental Encoding and Decoding 7.2.1.3.1. IncrementalEncoder Objects 7.2.1.3.2. IncrementalDecoder Objects 7.2.1.4. Stream Encoding and Decoding 7.2.1.4.1. StreamWriter Objects 7.2.1.4.2. StreamReader Objects 7.2.1.4.3. StreamReaderWriter Objects 7.2.1.4.4. StreamRecoder Objects 7.2.2. Encodings and Unicode 7.2.3. Standard Encodings 7.2.4. Python Specific Encodings 7.2.4.1. Text Encodings 7.2.4.2. Binary Transforms 7.2.4.3. Text Transforms 7.2.5. encodings.idna — Internationalized Domain Names in Applications 7.2.6. encodings.mbcs — Windows ANSI codepage 7.2.7. encodings.utf_8_sig — UTF-8 codec with BOM signature 8. Data Types 8.1. datetime — Basic date and time types 8.1.1. Available Types 8.1.2. timedelta Objects 8.1.3. date Objects 8.1.4. datetime Objects 8.1.5. time Objects 8.1.6. tzinfo Objects 8.1.7. timezone Objects 8.1.8. strftime() and strptime() Behavior 8.2. calendar — General calendar-related functions 8.3. collections — Container datatypes 8.3.1. ChainMap objects 8.3.1.1. ChainMap Examples and Recipes 8.3.2. Counter objects 8.3.3. deque objects 8.3.3.1. deque Recipes 8.3.4. defaultdict objects 8.3.4.1. defaultdict Examples 8.3.5. namedtuple() Factory Function for Tuples with Named Fields 8.3.6. OrderedDict objects 8.3.6.1. OrderedDict Examples and Recipes 8.3.7. UserDict objects 8.3.8. UserList objects 8.3.9. UserString objects 8.4. collections.abc — Abstract Base Classes for Containers 8.4.1. Collections Abstract Base Classes 8.5. heapq — Heap queue algorithm 8.5.1. Basic Examples 8.5.2. Priority Queue Implementation Notes 8.5.3. Theory 8.6. bisect — Array bisection algorithm 8.6.1. Searching Sorted Lists 8.6.2. Other Examples 8.7. array — Efficient arrays of numeric values 8.8. weakref — Weak references 8.8.1. Weak Reference Objects 8.8.2. Example 8.8.3. Finalizer Objects 8.8.4. Comparing finalizers with __del__() methods 8.9. types — Dynamic type creation and names for built-in types 8.9.1. Dynamic Type Creation 8.9.2. Standard Interpreter Types 8.9.3. Additional Utility Classes and Functions 8.9.4. Coroutine Utility Functions 8.10. copy — Shallow and deep copy operations 8.11. pprint — Data pretty printer 8.11.1. PrettyPrinter Objects 8.11.2. Example 8.12. reprlib — Alternate repr() implementation 8.12.1. Repr Objects 8.12.2. Subclassing Repr Objects 8.13. enum — Support for enumerations 8.13.1. Module Contents 8.13.2. Creating an Enum 8.13.3. Programmatic access to enumeration members and their attributes 8.13.4. Duplicating enum members and values 8.13.5. Ensuring unique enumeration values 8.13.6. Using automatic values 8.13.7. Iteration 8.13.8. Comparisons 8.13.9. Allowed members and attributes of enumerations 8.13.10. Restricted subclassing of enumerations 8.13.11. Pickling 8.13.12. Functional API 8.13.13. Derived Enumerations 8.13.13.1. IntEnum 8.13.13.2. IntFlag 8.13.13.3. Flag 8.13.13.4. Others 8.13.14. Interesting examples 8.13.14.1. Omitting values 8.13.14.1.1. Using auto 8.13.14.1.2. Using object 8.13.14.1.3. Using a descriptive string 8.13.14.1.4. Using a custom __new__() 8.13.14.2. OrderedEnum 8.13.14.3. DuplicateFreeEnum 8.13.14.4. Planet 8.13.15. How are Enums different? 8.13.15.1. Enum Classes 8.13.15.2. Enum Members (aka instances) 8.13.15.3. Finer Points 8.13.15.3.1. Supported __dunder__ names 8.13.15.3.2. Supported _sunder_ names 8.13.15.3.3. Enum member type 8.13.15.3.4. Boolean value of Enum classes and members 8.13.15.3.5. Enum classes with methods 8.13.15.3.6. Combining members of Flag 9. Numeric and Mathematical Modules 9.1. numbers — Numeric abstract base classes 9.1.1. The numeric tower 9.1.2. Notes for type implementors 9.1.2.1. Adding More Numeric ABCs 9.1.2.2. Implementing the arithmetic operations 9.2. math — Mathematical functions 9.2.1. Number-theoretic and representation functions 9.2.2. Power and logarithmic functions 9.2.3. Trigonometric functions 9.2.4. Angular conversion 9.2.5. Hyperbolic functions 9.2.6. Special functions 9.2.7. Constants 9.3. cmath — Mathematical functions for complex numbers 9.3.1. Conversions to and from polar coordinates 9.3.2. Power and logarithmic functions 9.3.3. Trigonometric functions 9.3.4. Hyperbolic functions 9.3.5. Classification functions 9.3.6. Constants 9.4. decimal — Decimal fixed point and floating point arithmetic 9.4.1. Quick-start Tutorial 9.4.2. Decimal objects 9.4.2.1. Logical operands 9.4.3. Context objects 9.4.4. Constants 9.4.5. Rounding modes 9.4.6. Signals 9.4.7. Floating Point Notes 9.4.7.1. Mitigating round-off error with increased precision 9.4.7.2. Special values 9.4.8. Working with threads 9.4.9. Recipes 9.4.10. Decimal FAQ 9.5. fractions — Rational numbers 9.6. random — Generate pseudo-random numbers 9.6.1. Bookkeeping functions 9.6.2. Functions for integers 9.6.3. Functions for sequences 9.6.4. Real-valued distributions 9.6.5. Alternative Generator 9.6.6. Notes on Reproducibility 9.6.7. Examples and Recipes 9.7. statistics — Mathematical statistics functions 9.7.1. Averages and measures of central location 9.7.2. Measures of spread 9.7.3. Function details 9.7.4. Exceptions 10. Functional Programming Modules 10.1. itertools — Functions creating iterators for efficient looping 10.1.1. Itertool functions 10.1.2. Itertools Recipes 10.2. functools — Higher-order functions and operations on callable objects 10.2.1. partial Objects 10.3. operator — Standard operators as functions 10.3.1. Mapping Operators to Functions 10.3.2. Inplace Operators 11. File and Directory Access 11.1. pathlib — Object-oriented filesystem paths 11.1.1. Basic use 11.1.2. Pure paths 11.1.2.1. General properties 11.1.2.2. Operators 11.1.2.3. Accessing individual parts 11.1.2.4. Methods and properties 11.1.3. Concrete paths 11.1.3.1. Methods 11.2. os.path — Common pathname manipulations 11.3. fileinput — Iterate over lines from multiple input streams 11.4. stat — Interpreting stat() results 11.5. filecmp — File and Directory Comparisons 11.5.1. The dircmp class 11.6. tempfile — Generate temporary files and directories 11.6.1. Examples 11.6.2. Deprecated functions and variables 11.7. glob — Unix style pathname pattern expansion 11.8. fnmatch — Unix filename pattern matching 11.9. linecache — Random access to text lines 11.10. shutil — High-level file operations 11.10.1. Directory and files operations 11.10.1.1. copytree example 11.10.1.2. rmtree example 11.10.2. Archiving operations 11.10.2.1. Archiving example 11.10.3. Querying the size of the output terminal 11.11. macpath — Mac OS 9 path manipulation functions 12. Data Persistence 12.1. pickle — Python object serialization 12.1.1. Relationship to other Python modules 12.1.1.1. Comparison with marshal 12.1.1.2. Comparison with json 12.1.2. Data stream format 12.1.3. Module Interface 12.1.4. What can be pickled and unpickled? 12.1.5. Pickling Class Instances 12.1.5.1. Persistence of External Objects 12.1.5.2. Dispatch Tables 12.1.5.3. Handling Stateful Objects 12.1.6. Restricting Globals 12.1.7. Performance 12.1.8. Examples 12.2. copyreg — Register pickle support functions 12.2.1. Example 12.3. shelve — Python object persistence 12.3.1. Restrictions 12.3.2. Example 12.4. marshal — Internal Python object serialization 12.5. dbm — Interfaces to Unix “databases” 12.5.1. dbm.gnu — GNU’s reinterpretation of dbm 12.5.2. dbm.ndbm — Interface based on ndbm 12.5.3. dbm.dumb — Portable DBM implementation 12.6. sqlite3 — DB-API 2.0 interface for SQLite databases 12.6.1. Module functions and constants 12.6.2. Connection Objects 12.6.3. Cursor Objects 12.6.4. Row Objects 12.6.5. Exceptions 12.6.6. SQLite and Python types 12.6.6.1. Introduction 12.6.6.2. Using adapters to store additional Python types in SQLite databases 12.6.6.2.1. Letting your object adapt itself 12.6.6.2.2. Registering an adapter callable 12.6.6.3. Converting SQLite values to custom Python types 12.6.6.4. Default adapters and converters 12.6.7. Controlling Transactions 12.6.8. Using sqlite3 efficiently 12.6.8.1. Using shortcut methods 12.6.8.2. Accessing columns by name instead of by index 12.6.8.3. Using the connection as a context manager 12.6.9. Common issues 12.6.9.1. Multithreading 13. Data Compression and Archiving 13.1. zlib — Compression compatible with gzip 13.2. gzip — Support for gzip files 13.2.1. Examples of usage 13.3. bz2 — Support for bzip2 compression 13.3.1. (De)compression of files 13.3.2. Incremental (de)compression 13.3.3. One-shot (de)compression 13.4. lzma — Compression using the LZMA algorithm 13.4.1. Reading and writing compressed files 13.4.2. Compressing and decompressing data in memory 13.4.3. Miscellaneous 13.4.4. Specifying custom filter chains 13.4.5. Examples 13.5. zipfile — Work with ZIP archives 13.5.1. ZipFile Objects 13.5.2. PyZipFile Objects 13.5.3. ZipInfo Objects 13.5.4. Command-Line Interface 13.5.4.1. Command-line options 13.6. tarfile — Read and write tar archive files 13.6.1. TarFile Objects 13.6.2. TarInfo Objects 13.6.3. Command-Line Interface 13.6.3.1. Command-line options 13.6.4. Examples 13.6.5. Supported tar formats 13.6.6. Unicode issues 14. File Formats 14.1. csv — CSV File Reading and Writing 14.1.1. Module Contents 14.1.2. Dialects and Formatting Parameters 14.1.3. Reader Objects 14.1.4. Writer Objects 14.1.5. Examples 14.2. configparser — Configuration file parser 14.2.1. Quick Start 14.2.2. Supported Datatypes 14.2.3. Fallback Values 14.2.4. Supported INI File Structure 14.2.5. Interpolation of values 14.2.6. Mapping Protocol Access 14.2.7. Customizing Parser Behaviour 14.2.8. Legacy API Examples 14.2.9. ConfigParser Objects 14.2.10. RawConfigParser Objects 14.2.11. Exceptions 14.3. netrc — netrc file processing 14.3.1. netrc Objects 14.4. xdrlib — Encode and decode XDR data 14.4.1. Packer Objects 14.4.2. Unpacker Objects 14.4.3. Exceptions 14.5. plistlib — Generate and parse Mac OS X .plist files 14.5.1. Examples 15. Cryptographic Services 15.1. hashlib — Secure hashes and message digests 15.1.1. Hash algorithms 15.1.2. SHAKE variable length digests 15.1.3. Key derivation 15.1.4. BLAKE2 15.1.4.1. Creating hash objects 15.1.4.2. Constants 15.1.4.3. Examples 15.1.4.3.1. Simple hashing 15.1.4.3.2. Using different digest sizes 15.1.4.3.3. Keyed hashing 15.1.4.3.4. Randomized hashing 15.1.4.3.5. Personalization 15.1.4.3.6. Tree mode 15.1.4.4. Credits 15.2. hmac — Keyed-Hashing for Message Authentication 15.3. secrets — Generate secure random numbers for managing secrets 15.3.1. Random numbers 15.3.2. Generating tokens 15.3.2.1. How many bytes should tokens use? 15.3.3. Other functions 15.3.4. Recipes and best practices 16. Generic Operating System Services 16.1. os — Miscellaneous operating system interfaces 16.1.1. File Names, Command Line Arguments, and Environment Variables 16.1.2. Process Parameters 16.1.3. File Object Creation 16.1.4. File Descriptor Operations 16.1.4.1. Querying the size of a terminal 16.1.4.2. Inheritance of File Descriptors 16.1.5. Files and Directories 16.1.5.1. Linux extended attributes 16.1.6. Process Management 16.1.7. Interface to the scheduler 16.1.8. Miscellaneous System Information 16.1.9. Random numbers 16.2. io — Core tools for working with streams 16.2.1. Overview 16.2.1.1. Text I/O 16.2.1.2. Binary I/O 16.2.1.3. Raw I/O 16.2.2. High-level Module Interface 16.2.2.1. In-memory streams 16.2.3. Class hierarchy 16.2.3.1. I/O Base Classes 16.2.3.2. Raw File I/O 16.2.3.3. Buffered Streams 16.2.3.4. Text I/O 16.2.4. Performance 16.2.4.1. Binary I/O 16.2.4.2. Text I/O 16.2.4.3. Multi-threading 16.2.4.4. Reentrancy 16.3. time — Time access and conversions 16.3.1. Functions 16.3.2. Clock ID Constants 16.3.3. Timezone Constants 16.4. argparse — Parser for command-line options, arguments and sub-commands 16.4.1. Example 16.4.1.1. Creating a parser 16.4.1.2. Adding arguments 16.4.1.3. Parsing arguments 16.4.2. ArgumentParser objects 16.4.2.1. prog 16.4.2.2. usage 16.4.2.3. description 16.4.2.4. epilog 16.4.2.5. parents 16.4.2.6. formatter_class 16.4.2.7. prefix_chars 16.4.2.8. fromfile_prefix_chars 16.4.2.9. argument_default 16.4.2.10. allow_abbrev 16.4.2.11. conflict_handler 16.4.2.12. add_help 16.4.3. The add_argument() method 16.4.3.1. name or flags 16.4.3.2. action 16.4.3.3. nargs 16.4.3.4. const 16.4.3.5. default 16.4.3.6. type 16.4.3.7. choices 16.4.3.8. required 16.4.3.9. help 16.4.3.10. metavar 16.4.3.11. dest 16.4.3.12. Action classes 16.4.4. The parse_args() method 16.4.4.1. Option value syntax 16.4.4.2. Invalid arguments 16.4.4.3. Arguments containing - 16.4.4.4. Argument abbreviations (prefix matching) 16.4.4.5. Beyond sys.argv 16.4.4.6. The Namespace object 16.4.5. Other utilities 16.4.5.1. Sub-commands 16.4.5.2. FileType objects 16.4.5.3. Argument groups 16.4.5.4. Mutual exclusion 16.4.5.5. Parser defaults 16.4.5.6. Printing help 16.4.5.7. Partial parsing 16.4.5.8. Customizing file parsing 16.4.5.9. Exiting methods 16.4.6. Upgrading optparse code 16.5. getopt — C-style parser for command line options 16.6. logging — Logging facility for Python 16.6.1. Logger Objects 16.6.2. Logging Levels 16.6.3. Handler Objects 16.6.4. Formatter Objects 16.6.5. Filter Objects 16.6.6. LogRecord Objects 16.6.7. LogRecord attributes 16.6.8. LoggerAdapter Objects 16.6.9. Thread Safety 16.6.10. Module-Level Functions 16.6.11. Module-Level Attributes 16.6.12. Integration with the warnings module 16.7. logging.config — Logging configuration 16.7.1. Configuration functions 16.7.2. Configuration dictionary schema 16.7.2.1. Dictionary Schema Details 16.7.2.2. Incremental Configuration 16.7.2.3. Object connections 16.7.2.4. User-defined objects 16.7.2.5. Access to external objects 16.7.2.6. Access to internal objects 16.7.2.7. Import resolution and custom importers 16.7.3. Configuration file format 16.8. logging.handlers — Logging handlers 16.8.1. StreamHandler 16.8.2. FileHandler 16.8.3. NullHandler 16.8.4. WatchedFileHandler 16.8.5. BaseRotatingHandler 16.8.6. RotatingFileHandler 16.8.7. TimedRotatingFileHandler 16.8.8. SocketHandler 16.8.9. DatagramHandler 16.8.10. SysLogHandler 16.8.11. NTEventLogHandler 16.8.12. SMTPHandler 16.8.13. MemoryHandler 16.8.14. HTTPHandler 16.8.15. QueueHandler 16.8.16. QueueListener 16.9. getpass — Portable password input 16.10. curses — Terminal handling for character-cell displays 16.10.1. Functions 16.10.2. Window Objects 16.10.3. Constants 16.11. curses.textpad — Text input widget for curses programs 16.11.1. Textbox objects 16.12. curses.ascii — Utilities for ASCII characters 16.13. curses.panel — A panel stack extension for curses 16.13.1. Functions 16.13.2. Panel Objects 16.14. platform — Access to underlying platform’s identifying data 16.14.1. Cross Platform 16.14.2. Java Platform 16.14.3. Windows Platform 16.14.3.1. Win95/98 specific 16.14.4. Mac OS Platform 16.14.5. Unix Platforms 16.15. errno — Standard errno system symbols 16.16. ctypes — A foreign function library for Python 16.16.1. ctypes tutorial 16.16.1.1. Loading dynamic link libraries 16.16.1.2. Accessing functions from loaded dlls 16.16.1.3. Calling functions 16.16.1.4. Fundamental data types 16.16.1.5. Calling functions, continued 16.16.1.6. Calling functions with your own custom data types 16.16.1.7. Specifying the required argument types (function prototypes) 16.16.1.8. Return types 16.16.1.9. Passing pointers (or: passing parameters by reference) 16.16.1.10. Structures and unions 16.16.1.11. Structure/union alignment and byte order 16.16.1.12. Bit fields in structures and unions 16.16.1.13. Arrays 16.16.1.14. Pointers 16.16.1.15. Type conversions 16.16.1.16. Incomplete Types 16.16.1.17. Callback functions 16.16.1.18. Accessing values exported from dlls 16.16.1.19. Surprises 16.16.1.20. Variable-sized data types 16.16.2. ctypes reference 16.16.2.1. Finding shared libraries 16.16.2.2. Loading shared libraries 16.16.2.3. Foreign functions 16.16.2.4. Function prototypes 16.16.2.5. Utility functions 16.16.2.6. Data types 16.16.2.7. Fundamental data types 16.16.2.8. Structured data types 16.16.2.9. Arrays and pointers 17. Concurrent Execution 17.1. threading — Thread-based parallelism 17.1.1. Thread-Local Data 17.1.2. Thread Objects 17.1.3. Lock Objects 17.1.4. RLock Objects 17.1.5. Condition Objects 17.1.6. Semaphore Objects 17.1.6.1. Semaphore Example 17.1.7. Event Objects 17.1.8. Timer Objects 17.1.9. Barrier Objects 17.1.10. Using locks, conditions, and semaphores in the with statement 17.2. multiprocessing — Process-based parallelism 17.2.1. Introduction 17.2.1.1. The Process class 17.2.1.2. Contexts and start methods 17.2.1.3. Exchanging objects between processes 17.2.1.4. Synchronization between processes 17.2.1.5. Sharing state between processes 17.2.1.6. Using a pool of workers 17.2.2. Reference 17.2.2.1. Process and exceptions 17.2.2.2. Pipes and Queues 17.2.2.3. Miscellaneous 17.2.2.4. Connection Objects 17.2.2.5. Synchronization primitives 17.2.2.6. Shared ctypes Objects 17.2.2.6.1. The multiprocessing.sharedctypes module 17.2.2.7. Managers 17.2.2.7.1. Customized managers 17.2.2.7.2. Using a remote manager 17.2.2.8. Proxy Objects 17.2.2.8.1. Cleanup 17.2.2.9. Process Pools 17.2.2.10. Listeners and Clients 17.2.2.10.1. Address Formats 17.2.2.11. Authentication keys 17.2.2.12. Logging 17.2.2.13. The multiprocessing.dummy module 17.2.3. Programming guidelines 17.2.3.1. All start methods 17.2.3.2. The spawn and forkserver start methods 17.2.4. Examples 17.3. The concurrent package 17.4. concurrent.futures — Launching parallel tasks 17.4.1. Executor Objects 17.4.2. ThreadPoolExecutor 17.4.2.1. ThreadPoolExecutor Example 17.4.3. ProcessPoolExecutor 17.4.3.1. ProcessPoolExecutor Example 17.4.4. Future Objects 17.4.5. Module Functions 17.4.6. Exception classes 17.5. subprocess — Subprocess management 17.5.1. Using the subprocess Module 17.5.1.1. Frequently Used Arguments 17.5.1.2. Popen Constructor 17.5.1.3. Exceptions 17.5.2. Security Considerations 17.5.3. Popen Objects 17.5.4. Windows Popen Helpers 17.5.4.1. Constants 17.5.5. Older high-level API 17.5.6. Replacing Older Functions with the subprocess Module 17.5.6.1. Replacing /bin/sh shell backquote 17.5.6.2. Replacing shell pipeline 17.5.6.3. Replacing os.system() 17.5.6.4. Replacing the os.spawn family 17.5.6.5. Replacing os.popen(), os.popen2(), os.popen3() 17.5.6.6. Replacing functions from the popen2 module 17.5.7. Legacy Shell Invocation Functions 17.5.8. Notes 17.5.8.1. Converting an argument sequence to a string on Windows 17.6. sched — Event scheduler 17.6.1. Scheduler Objects 17.7. queue — A synchronized queue class 17.7.1. Queue Objects 17.8. dummy_threading — Drop-in replacement for the threading module 17.9. _thread — Low-level threading API 17.10. _dummy_thread — Drop-in replacement for the _thread module 18. Interprocess Communication and Networking 18.1. socket — Low-level networking interface 18.1.1. Socket families 18.1.2. Module contents 18.1.2.1. Exceptions 18.1.2.2. Constants 18.1.2.3. Functions 18.1.2.3.1. Creating sockets 18.1.2.3.2. Other functions 18.1.3. Socket Objects 18.1.4. Notes on socket timeouts 18.1.4.1. Timeouts and the connect method 18.1.4.2. Timeouts and the accept method 18.1.5. Example 18.2. ssl — TLS/SSL wrapper for socket objects 18.2.1. Functions, Constants, and Exceptions 18.2.1.1. Socket creation 18.2.1.2. Context creation 18.2.1.3. Random generation 18.2.1.4. Certificate handling 18.2.1.5. Constants 18.2.2. SSL Sockets 18.2.3. SSL Contexts 18.2.4. Certificates 18.2.4.1. Certificate chains 18.2.4.2. CA certificates 18.2.4.3. Combined key and certificate 18.2.4.4. Self-signed certificates 18.2.5. Examples 18.2.5.1. Testing for SSL support 18.2.5.2. Client-side operation 18.2.5.3. Server-side operation 18.2.6. Notes on non-blocking sockets 18.2.7. Memory BIO Support 18.2.8. SSL session 18.2.9. Security considerations 18.2.9.1. Best defaults 18.2.9.2. Manual settings 18.2.9.2.1. Verifying certificates 18.2.9.2.2. Protocol versions 18.2.9.2.3. Cipher selection 18.2.9.3. Multi-processing 18.2.10. LibreSSL support 18.3. select — Waiting for I/O completion 18.3.1. /dev/poll Polling Objects 18.3.2. Edge and Level Trigger Polling (epoll) Objects 18.3.3. Polling Objects 18.3.4. Kqueue Objects 18.3.5. Kevent Objects 18.4. selectors — High-level I/O multiplexing 18.4.1. Introduction 18.4.2. Classes 18.4.3. Examples 18.5. asyncio — Asynchronous I/O, event loop, coroutines and tasks 18.5.1. Base Event Loop 18.5.1.1. Run an event loop 18.5.1.2. Calls 18.5.1.3. Delayed calls 18.5.1.4. Futures 18.5.1.5. Tasks 18.5.1.6. Creating connections 18.5.1.7. Creating listening connections 18.5.1.8. Watch file descriptors 18.5.1.9. Low-level socket operations 18.5.1.10. Resolve host name 18.5.1.11. Connect pipes 18.5.1.12. UNIX signals 18.5.1.13. Executor 18.5.1.14. Error Handling API 18.5.1.15. Debug mode 18.5.1.16. Server 18.5.1.17. Handle 18.5.1.18. Event loop examples 18.5.1.18.1. Hello World with call_soon() 18.5.1.18.2. Display the current date with call_later() 18.5.1.18.3. Watch a file descriptor for read events 18.5.1.18.4. Set signal handlers for SIGINT and SIGTERM 18.5.2. Event loops 18.5.2.1. Event loop functions 18.5.2.2. Available event loops 18.5.2.3. Platform support 18.5.2.3.1. Windows 18.5.2.3.2. Mac OS X 18.5.2.4. Event loop policies and the default policy 18.5.2.5. Event loop policy interface 18.5.2.6. Access to the global loop policy 18.5.2.7. Customizing the event loop policy 18.5.3. Tasks and coroutines 18.5.3.1. Coroutines 18.5.3.1.1. Example: Hello World coroutine 18.5.3.1.2. Example: Coroutine displaying the current date 18.5.3.1.3. Example: Chain coroutines 18.5.3.2. InvalidStateError 18.5.3.3. TimeoutError 18.5.3.4. Future 18.5.3.4.1. Example: Future with run_until_complete() 18.5.3.4.2. Example: Future with run_forever() 18.5.3.5. Task 18.5.3.5.1. Example: Parallel execution of tasks 18.5.3.6. Task functions 18.5.4. Transports and protocols (callback based API) 18.5.4.1. Transports 18.5.4.1.1. BaseTransport 18.5.4.1.2. ReadTransport 18.5.4.1.3. WriteTransport 18.5.4.1.4. DatagramTransport 18.5.4.1.5. BaseSubprocessTransport 18.5.4.2. Protocols 18.5.4.2.1. Protocol classes 18.5.4.2.2. Connection callbacks 18.5.4.2.3. Streaming protocols 18.5.4.2.4. Datagram protocols 18.5.4.2.5. Flow control callbacks 18.5.4.2.6. Coroutines and protocols 18.5.4.3. Protocol examples 18.5.4.3.1. TCP echo client protocol 18.5.4.3.2. TCP echo server protocol 18.5.4.3.3. UDP echo client protocol 18.5.4.3.4. UDP echo server protocol 18.5.4.3.5. Register an open socket to wait for data using a protocol 18.5.5. Streams (coroutine based API) 18.5.5.1. Stream functions 18.5.5.2. StreamReader 18.5.5.3. StreamWriter 18.5.5.4. StreamReaderProtocol 18.5.5.5. IncompleteReadError 18.5.5.6. LimitOverrunError 18.5.5.7. Stream examples 18.5.5.7.1. TCP echo client using streams 18.5.5.7.2. TCP echo server using streams 18.5.5.7.3. Get HTTP headers 18.5.5.7.4. Register an open socket to wait for data using streams 18.5.6. Subprocess 18.5.6.1. Windows event loop 18.5.6.2. Create a subprocess: high-level API using Process 18.5.6.3. Create a subprocess: low-level API using subprocess.Popen 18.5.6.4. Constants 18.5.6.5. Process 18.5.6.6. Subprocess and threads 18.5.6.7. Subprocess examples 18.5.6.7.1. Subprocess using transport and protocol 18.5.6.7.2. Subprocess using streams 18.5.7. Synchronization primitives 18.5.7.1. Locks 18.5.7.1.1. Lock 18.5.7.1.2. Event 18.5.7.1.3. Condition 18.5.7.2. Semaphores 18.5.7.2.1. Semaphore 18.5.7.2.2. BoundedSemaphore 18.5.8. Queues 18.5.8.1. Queue 18.5.8.2. PriorityQueue 18.5.8.3. LifoQueue 18.5.8.3.1. Exceptions 18.5.9. Develop with asyncio 18.5.9.1. Debug mode of asyncio 18.5.9.2. Cancellation 18.5.9.3. Concurrency and multithreading 18.5.9.4. Handle blocking functions correctly 18.5.9.5. Logging 18.5.9.6. Detect coroutine objects never scheduled 18.5.9.7. Detect exceptions never consumed 18.5.9.8. Chain coroutines correctly 18.5.9.9. Pending task destroyed 18.5.9.10. Close transports and event loops 18.6. asyncore — Asynchronous socket handler 18.6.1. asyncore Example basic HTTP client 18.6.2. asyncore Example basic echo server 18.7. asynchat — Asynchronous socket command/response handler 18.7.1. asynchat Example 18.8. signal — Set handlers for asynchronous events 18.8.1. General rules 18.8.1.1. Execution of Python signal handlers 18.8.1.2. Signals and threads 18.8.2. Module contents 18.8.3. Example 18.9. mmap — Memory-mapped file support 19. Internet Data Handling 19.1. email — An email and MIME handling package 19.1.1. email.message: Representing an email message 19.1.2. email.parser: Parsing email messages 19.1.2.1. FeedParser API 19.1.2.2. Parser API 19.1.2.3. Additional notes 19.1.3. email.generator: Generating MIME documents 19.1.4. email.policy: Policy Objects 19.1.5. email.errors: Exception and Defect classes 19.1.6. email.headerregistry: Custom Header Objects 19.1.7. email.contentmanager: Managing MIME Content 19.1.7.1. Content Manager Instances 19.1.8. email: Examples 19.1.9. email.message.Message: Representing an email message using the compat32 API 19.1.10. email.mime: Creating email and MIME objects from scratch 19.1.11. email.header: Internationalized headers 19.1.12. email.charset: Representing character sets 19.1.13. email.encoders: Encoders 19.1.14. email.utils: Miscellaneous utilities 19.1.15. email.iterators: Iterators 19.2. json — JSON encoder and decoder 19.2.1. Basic Usage 19.2.2. Encoders and Decoders 19.2.3. Exceptions 19.2.4. Standard Compliance and Interoperability 19.2.4.1. Character Encodings 19.2.4.2. Infinite and NaN Number Values 19.2.4.3. Repeated Names Within an Object 19.2.4.4. Top-level Non-Object, Non-Array Values 19.2.4.5. Implementation Limitations 19.2.5. Command Line Interface 19.2.5.1. Command line options 19.3. mailcap — Mailcap file handling 19.4. mailbox — Manipulate mailboxes in various formats 19.4.1. Mailbox objects 19.4.1.1. Maildir 19.4.1.2. mbox 19.4.1.3. MH 19.4.1.4. Babyl 19.4.1.5. MMDF 19.4.2. Message objects 19.4.2.1. MaildirMessage 19.4.2.2. mboxMessage 19.4.2.3. MHMessage 19.4.2.4. BabylMessage 19.4.2.5. MMDFMessage 19.4.3. Exceptions 19.4.4. Examples 19.5. mimetypes — Map filenames to MIME types 19.5.1. MimeTypes Objects 19.6. base64 — Base16, Base32, Base64, Base85 Data Encodings 19.7. binhex — Encode and decode binhex4 files 19.7.1. Notes 19.8. binascii — Convert between binary and ASCII 19.9. quopri — Encode and decode MIME quoted-printable data 19.10. uu — Encode and decode uuencode files 20. Structured Markup Processing Tools 20.1. html — HyperText Markup Language support 20.2. html.parser — Simple HTML and XHTML parser 20.2.1. Example HTML Parser Application 20.2.2. HTMLParser Methods 20.2.3. Examples 20.3. html.entities — Definitions of HTML general entities 20.4. XML Processing Modules 20.4.1. XML vulnerabilities 20.4.2. The defusedxml and defusedexpat Packages 20.5. xml.etree.ElementTree — The ElementTree XML API 20.5.1. Tutorial 20.5.1.1. XML tree and elements 20.5.1.2. Parsing XML 20.5.1.3. Pull API for non-blocking parsing 20.5.1.4. Finding interesting elements 20.5.1.5. Modifying an XML File 20.5.1.6. Building XML documents 20.5.1.7. Parsing XML with Namespaces 20.5.1.8. Additional resources 20.5.2. XPath support 20.5.2.1. Example 20.5.2.2. Supported XPath syntax 20.5.3. Reference 20.5.3.1. Functions 20.5.3.2. Element Objects 20.5.3.3. ElementTree Objects 20.5.3.4. QName Objects 20.5.3.5. TreeBuilder Objects 20.5.3.6. XMLParser Objects 20.5.3.7. XMLPullParser Objects 20.5.3.8. Exceptions 20.6. xml.dom — The Document Object Model API 20.6.1. Module Contents 20.6.2. Objects in the DOM 20.6.2.1. DOMImplementation Objects 20.6.2.2. Node Objects 20.6.2.3. NodeList Objects 20.6.2.4. DocumentType Objects 20.6.2.5. Document Objects 20.6.2.6. Element Objects 20.6.2.7. Attr Objects 20.6.2.8. NamedNodeMap Objects 20.6.2.9. Comment Objects 20.6.2.10. Text and CDATASection Objects 20.6.2.11. ProcessingInstruction Objects 20.6.2.12. Exceptions 20.6.3. Conformance 20.6.3.1. Type Mapping 20.6.3.2. Accessor Methods 20.7. xml.dom.minidom — Minimal DOM implementation 20.7.1. DOM Objects 20.7.2. DOM Example 20.7.3. minidom and the DOM standard 20.8. xml.dom.pulldom — Support for building partial DOM trees 20.8.1. DOMEventStream Objects 20.9. xml.sax — Support for SAX2 parsers 20.9.1. SAXException Objects 20.10. xml.sax.handler — Base classes for SAX handlers 20.10.1. ContentHandler Objects 20.10.2. DTDHandler Objects 20.10.3. EntityResolver Objects 20.10.4. ErrorHandler Objects 20.11. xml.sax.saxutils — SAX Utilities 20.12. xml.sax.xmlreader — Interface for XML parsers 20.12.1. XMLReader Objects 20.12.2. IncrementalParser Objects 20.12.3. Locator Objects 20.12.4. InputSource Objects 20.12.5. The Attributes Interface 20.12.6. The AttributesNS Interface 20.13. xml.parsers.expat — Fast XML parsing using Expat 20.13.1. XMLParser Objects 20.13.2. ExpatError Exceptions 20.13.3. Example 20.13.4. Content Model Descriptions 20.13.5. Expat error constants 21. Internet Protocols and Support 21.1. webbrowser — Convenient Web-browser controller 21.1.1. Browser Controller Objects 21.2. cgi — Common Gateway Interface support 21.2.1. Introduction 21.2.2. Using the cgi module 21.2.3. Higher Level Interface 21.2.4. Functions 21.2.5. Caring about security 21.2.6. Installing your CGI script on a Unix system 21.2.7. Testing your CGI script 21.2.8. Debugging CGI scripts 21.2.9. Common problems and solutions 21.3. cgitb — Traceback manager for CGI scripts 21.4. wsgiref — WSGI Utilities and Reference Implementation 21.4.1. wsgiref.util – WSGI environment utilities 21.4.2. wsgiref.headers – WSGI response header tools 21.4.3. wsgiref.simple_server – a simple WSGI HTTP server 21.4.4. wsgiref.validate — WSGI conformance checker 21.4.5. wsgiref.handlers – server/gateway base classes 21.4.6. Examples 21.5. urllib — URL handling modules 21.6. urllib.request — Extensible library for opening URLs 21.6.1. Request Objects 21.6.2. OpenerDirector Objects 21.6.3. BaseHandler Objects 21.6.4. HTTPRedirectHandler Objects 21.6.5. HTTPCookieProcessor Objects 21.6.6. ProxyHandler Objects 21.6.7. HTTPPasswordMgr Objects 21.6.8. HTTPPasswordMgrWithPriorAuth Objects 21.6.9. AbstractBasicAuthHandler Objects 21.6.10. HTTPBasicAuthHandler Objects 21.6.11. ProxyBasicAuthHandler Objects 21.6.12. AbstractDigestAuthHandler Objects 21.6.13. HTTPDigestAuthHandler Objects 21.6.14. ProxyDigestAuthHandler Objects 21.6.15. HTTPHandler Objects 21.6.16. HTTPSHandler Objects 21.6.17. FileHandler Objects 21.6.18. DataHandler Objects 21.6.19. FTPHandler Objects 21.6.20. CacheFTPHandler Objects 21.6.21. UnknownHandler Objects 21.6.22. HTTPErrorProcessor Objects 21.6.23. Examples 21.6.24. Legacy interface 21.6.25. urllib.request Restrictions 21.7. urllib.response — Response classes used by urllib 21.8. urllib.parse — Parse URLs into components 21.8.1. URL Parsing 21.8.2. Parsing ASCII Encoded Bytes 21.8.3. Structured Parse Results 21.8.4. URL Quoting 21.9. urllib.error — Exception classes raised by urllib.request 21.10. urllib.robotparser — Parser for robots.txt 21.11. http — HTTP modules 21.11.1. HTTP status codes 21.12. http.client — HTTP protocol client 21.12.1. HTTPConnection Objects 21.12.2. HTTPResponse Objects 21.12.3. Examples 21.12.4. HTTPMessage Objects 21.13. ftplib — FTP protocol client 21.13.1. FTP Objects 21.13.2. FTP_TLS Objects 21.14. poplib — POP3 protocol client 21.14.1. POP3 Objects 21.14.2. POP3 Example 21.15. imaplib — IMAP4 protocol client 21.15.1. IMAP4 Objects 21.15.2. IMAP4 Example 21.16. nntplib — NNTP protocol client 21.16.1. NNTP Objects 21.16.1.1. Attributes 21.16.1.2. Methods 21.16.2. Utility functions 21.17. smtplib — SMTP protocol client 21.17.1. SMTP Objects 21.17.2. SMTP Example 21.18. smtpd — SMTP Server 21.18.1. SMTPServer Objects 21.18.2. DebuggingServer Objects 21.18.3. PureProxy Objects 21.18.4. MailmanProxy Objects 21.18.5. SMTPChannel Objects 21.19. telnetlib — Telnet client 21.19.1. Telnet Objects 21.19.2. Telnet Example 21.20. uuid — UUID objects according to RFC 4122 21.20.1. Example 21.21. socketserver — A framework for network servers 21.21.1. Server Creation Notes 21.21.2. Server Objects 21.21.3. Request Handler Objects 21.21.4. Examples 21.21.4.1. socketserver.TCPServer Example 21.21.4.2. socketserver.UDPServer Example 21.21.4.3. Asynchronous Mixins 21.22. http.server — HTTP servers 21.23. http.cookies — HTTP state management 21.23.1. Cookie Objects 21.23.2. Morsel Objects 21.23.3. Example 21.24. http.cookiejar — Cookie handling for HTTP clients 21.24.1. CookieJar and FileCookieJar Objects 21.24.2. FileCookieJar subclasses and co-operation with web browsers 21.24.3. CookiePolicy Objects 21.24.4. DefaultCookiePolicy Objects 21.24.5. Cookie Objec
Contents Overview 1 Lesson 1: Concepts – Locks and Lock Manager 3 Lesson 2: Concepts – Batch and Transaction 31 Lesson 3: Concepts – Locks and Applications 51 Lesson 4: Information Collection and Analysis 63 Lesson 5: Concepts – Formulating and Implementing Resolution 81 Module 4: Troubleshooting Locking and Blocking Overview At the end of this module, you will be able to:  Discuss how lock manager uses lock mode, lock resources, and lock compatibility to achieve transaction isolation.  Describe the various transaction types and how transactions differ from batches.  Describe how to troubleshoot blocking and locking issues.  Analyze the output of blocking scripts and Microsoft® SQL Server™ Profiler to troubleshoot locking and blocking issues.  Formulate hypothesis to resolve locking and blocking issues. Lesson 1: Concepts – Locks and Lock Manager This lesson outlines some of the common causes that contribute to the perception of a slow server. What You Will Learn After completing this lesson, you will be able to:  Describe locking architecture used by SQL Server.  Identify the various lock modes used by SQL Server.  Discuss lock compatibility and concurrent access.  Identify different types of lock resources.  Discuss dynamic locking and lock escalation.  Differentiate locks, latches, and other SQL Server internal “locking” mechanism such as spinlocks and other synchronization objects. Recommended Reading  Chapter 14 “Locking”, Inside SQL Server 2000 by Kalen Delaney  SOX000821700049 – SQL 7.0 How to interpret lock resource Ids  SOX000925700237 – TITLE: Lock escalation in SQL 7.0  SOX001109700040 – INF: Queries with PREFETCH in the plan hold lock until the end of transaction Locking Concepts Delivery Tip Prior to delivering this material, test the class to see if they fully understand the different isolation levels. If the class is not confident in their understanding, review appendix A04_Locking and its accompanying PowerPoint® file. Transactions in SQL Server provide the ACID properties: Atomicity A transaction either commits or aborts. If a transaction commits, all of its effects remain. If it aborts, all of its effects are undone. It is an “all or nothing” operation. Consistency An application should maintain the consistency of a database. For example, if you defer constraint checking, it is your responsibility to ensure that the database is consistent. Isolation Concurrent transactions are isolated from the updates of other incomplete transactions. These updates do not constitute a consistent state. This property is often called serializability. For example, a second transaction traversing the doubly linked list mentioned above would see the list before or after the insert, but it will see only complete changes. Durability After a transaction commits, its effects will persist even if there are system failures. Consistency and isolation are the most important in describing SQL Server’s locking model. It is up to the application to define what consistency means, and isolation in some form is needed to achieve consistent results. SQL Server uses locking to achieve isolation. Definition of Dependency: A set of transactions can run concurrently if their outputs are disjoint from the union of one another’s input and output sets. For example, if T1 writes some object that is in T2’s input or output set, there is a dependency between T1 and T2. Bad Dependencies These include lost updates, dirty reads, non-repeatable reads, and phantoms. ANSI SQL Isolation Levels An isolation level determines the degree to which data is isolated for use by one process and guarded against interference from other processes. Prior to SQL Server 7.0, REPEATABLE READ and SERIALIZABLE isolation levels were synonymous. There was no way to prevent non-repeatable reads while not preventing phantoms. By default, SQL Server 2000 operates at an isolation level of READ COMMITTED. To make use of either more or less strict isolation levels in applications, locking can be customized for an entire session by setting the isolation level of the session with the SET TRANSACTION ISOLATION LEVEL statement. To determine the transaction isolation level currently set, use the DBCC USEROPTIONS statement, for example: USE pubs GO SET TRANSACTION ISOLATION LEVEL REPEATABLE READ GO DBCC USEROPTIONS GO Multigranular Locking Multigranular Locking In our example, if one transaction (T1) holds an exclusive lock at the table level, and another transaction (T2) holds an exclusive lock at the row level, each of the transactions believe they have exclusive access to the resource. In this scenario, since T1 believes it locks the entire table, it might inadvertently make changes to the same row that T2 thought it has locked exclusively. In a multigranular locking environment, there must be a way to effectively overcome this scenario. Intent lock is the answer to this problem. Intent Lock Intent Lock is the term used to mean placing a marker in a higher-level lock queue. The type of intent lock can also be called the multigranular lock mode. An intent lock indicates that SQL Server wants to acquire a shared (S) lock or exclusive (X) lock on some of the resources lower down in the hierarchy. For example, a shared intent lock placed at the table level means that a transaction intends on placing shared (S) locks on pages or rows within that table. Setting an intent lock at the table level prevents another transaction from subsequently acquiring an exclusive (X) lock on the table containing that page. Intent locks improve performance because SQL Server examines intent locks only at the table level to determine whether a transaction can safely acquire a lock on that table. This removes the requirement to examine every row or page lock on the table to determine whether a transaction can lock the entire table. Lock Mode The code shown in the slide represents how the lock mode is stored internally. You can see these codes by querying the master.dbo.spt_values table: SELECT * FROM master.dbo.spt_values WHERE type = N'L' However, the req_mode column of master.dbo.syslockinfo has lock mode code that is one less than the code values shown here. For example, value of req_mode = 3 represents the Shared lock mode rather than the Schema Modification lock mode. Lock Compatibility These locks can apply at any coarser level of granularity. If a row is locked, SQL Server will apply intent locks at both the page and the table level. If a page is locked, SQL Server will apply an intent lock at the table level. SIX locks imply that we have shared access to a resource and we have also placed X locks at a lower level in the hierarchy. SQL Server never asks for SIX locks directly, they are always the result of a conversion. For example, suppose a transaction scanned a page using an S lock and then subsequently decided to perform a row level update. The row would obtain an X lock, but now the page would require an IX lock. The resultant mode on the page would be SIX. Another type of table lock is a schema stability lock (Sch-S) and is compatible with all table locks except the schema modification lock (Sch-M). The schema modification lock (Sch-M) is incompatible with all table locks. Locking Resources Delivery Tip Note the differences between Key and Key Range locks. Key Range locks will be covered in a couple of slides. SQL Server can lock these resources: Item Description DB A database. File A database file Index An entire index of a table. Table An entire table, including all data and indexes. Extent A contiguous group of data pages or index pages. Page An 8-KB data page or index page. Key Row lock within an index. Key-range A key-range. Used to lock ranges between records in a table to prevent phantom insertions or deletions into a set of records. Ensures serializable transactions. RID A Row Identifier. Used to individually lock a single row within a table. Application A lock resource defined by an application. The lock manager knows nothing about the resource format. It simply compares the 'strings' representing the lock resources to determine whether it has found a match. If a match is found, it knows that resource is already locked. Some of the resources have “sub-resources.” The followings are sub-resources displayed by the sp_lock output: Database Lock Sub-Resources: Full Database Lock (default) [BULK-OP-DB] – Bulk Operation Lock for Database [BULK-OP-LOG] – Bulk Operation Lock for Log Table Lock Sub-Resources: Full Table Lock (default) [UPD-STATS] – Update statistics Lock [COMPILE] – Compile Lock Index Lock sub-Resources: Full Index Lock (default) [INDEX_ID] – Index ID Lock [INDEX_NAME] – Index Name Lock [BULK_ALLOC] – Bulk Allocation Lock [DEFRAG] – Defragmentation Lock For more information, see also… SOX000821700049 SQL 7.0 How to interpret lock resource Ids Lock Resource Block The resource type has the following resource block format: Resource Type (Code) Content DB (2) Data 1: sub-resource; Data 2: 0; Data 3: 0 File (3) Data 1: File ID; Data 2: 0; Data 3: 0 Index (4) Data 1: Object ID; Data 2: sub-resource; Data 3: Index ID Table (5) Data 1: Object ID; Data 2: sub-resource; Data 3: 0. Page (6) Data 1: Page Number; Data 3: 0. Key (7) Data 1: Object ID; Data 2: Index ID; Data 3: Hashed Key Extent (8) Data 1: Extent ID; Data 3: 0. RID (9) Data 1: RID; Data 3: 0. Application (10) Data 1: Application resource name The rsc_bin column of master..syslockinfo contains the resource block in hexadecimal format. For an example of how to decode value from this column using the information above, let us assume we have the following value: 0x000705001F83D775010002014F0BEC4E With byte swapping within each field, this can be decoded as: Byte 0: Flag – 0x00 Byte 1: Resource Type – 0x07 (Key) Byte 2-3: DBID – 0x0005 Byte 4-7: ObjectID – 0x 75D7831F (1977058079) Byte 8-9: IndexID – 0x0001 Byte 10-16: Hash Key value – 0x 02014F0BEC4E For more information about how to decode this value, see also… Inside SQL Server 2000, pages 803 and 806. Key Range Locking Key Range Locking To support SERIALIZABLE transaction semantics, SQL Server needs to lock sets of rows specified by a predicate, such as WHERE salary BETWEEN 30000 AND 50000 SQL Server needs to lock data that does not exist! If no rows satisfy the WHERE condition the first time the range is scanned, no rows should be returned on any subsequent scans. Key range locks are similar to row locks on index keys (whether clustered or not). The locks are placed on individual keys rather than at the node level. The hash value consists of all the key components and the locator. So, for a nonclustered index over a heap, where columns c1 and c2 where indexed, the hash would contain contributions from c1, c2 and the RID. A key range lock applied to a particular key means that all keys between the value locked and the next value would be locked for all data modification. Key range locks can lock a slightly larger range than that implied by the WHERE clause. Suppose the following select was executed in a transaction with isolation level SERIALIZABLE: SELECT * FROM members WHERE first_name between ‘Al’ and ‘Carl’ If 'Al', 'Bob', and 'Dave' are index keys in the table, the first two of these would acquire key range locks. Although this would prevent anyone from inserting either 'Alex' or 'Ben', it would also prevent someone from inserting 'Dan', which is not within the range of the WHERE clause. Prior to SQL Server 7.0, page locking was used to prevent phantoms by locking the entire set of pages on which the phantom would exist. This can be too conservative. Key Range locking lets SQL Server lock only a much more restrictive area of the table. Impact Key-range locking ensures that these scenarios are SERIALIZABLE:  Range scan query  Singleton fetch of nonexistent row  Delete operation  Insert operation However, the following conditions must be satisfied before key-range locking can occur:  The transaction-isolation level must be set to SERIALIZABLE.  The operation performed on the data must use an index range access. Range locking is activated only when query processing (such as the optimizer) chooses an index path to access the data. Key Range Lock Mode Again, the req_mode column of master.dbo.syslockinfo has lock mode code that is one less than the code values shown here. Dynamic Locking When modifying individual rows, SQL Server typically would take row locks to maximize concurrency (for example, OLTP, order-entry application). When scanning larger volumes of data, it would be more appropriate to take page or table locks to minimize the cost of acquiring locks (for example, DSS, data warehouse, reporting). Locking Decision The decision about which unit to lock is made dynamically, taking many factors into account, including other activity on the system. For example, if there are multiple transactions currently accessing a table, SQL Server will tend to favor row locking more so than it otherwise would. It may mean the difference between scanning the table now and paying a bit more in locking cost, or having to wait to acquire a more coarse lock. A preliminary locking decision is made during query optimization, but that decision can be adjusted when the query is actually executed. Lock Escalation When the lock count for the transaction exceeds and is a multiple of ESCALATION_THRESHOLD (1250), the Lock Manager attempts to escalate. For example, when a transaction acquired 1250 locks, lock manager will try to escalate. The number of locks held may continue to increase after the escalation attempt (for example, because new tables are accessed, or the previous lock escalation attempts failed due to incompatible locks held by another spid). If the lock count for this transaction reaches 2500 (1250 * 2), Lock Manager will attempt escalation again. The Lock Manager looks at the lock memory it is using and if it is more than 40 percent of SQL Server’s allocated buffer pool memory, it tries to find a scan (SDES) where no escalation has already been performed. It then repeats the search operation until all scans have been escalated or until the memory used drops under the MEMORY_LOAD_ESCALATION_THRESHOLD (40%) value. If lock escalation is not possible or fails to significantly reduce lock memory footprint, SQL Server can continue to acquire locks until the total lock memory reaches 60 percent of the buffer pool (MAX_LOCK_RESOURCE_MEMORY_PERCENTAGE=60). Lock escalation may be also done when a single scan (SDES) holds more than LOCK_ESCALATION_THRESHOLD (765) locks. There is no lock escalation on temporary tables or system tables. Trace Flag 1211 disables lock escalation. Important Do not relay this to the customer without careful consideration. Lock escalation is a necessary feature, not something to be avoided completely. Trace flags are global and disabling lock escalation could lead to out of memory situations, extremely poor performing queries, or other problems. Lock escalation tracing can be seen using the Profiler or with the general locking trace flag, -T1200. However, Trace Flag 1200 shows all lock activity so it should not be usable on a production system. For more information, see also… SOX000925700237 “TITLE: SQL 7.0 Lock escalation in SQL 7.0” Lock Timeout Application Lock Timeout An application can set lock timeout for a session with the SET option: SET LOCK_TIMEOUT N where N is a number of milliseconds. A value of -1 means that there will be no timeout, which is equivalent to the version 6.5 behavior. A value of 0 means that there will be no waiting; if a process finds a resource locked, it will generate error message 1222 and continue with the next statement. The current value of LOCK_TIMEOUT is stored in the global variable @@lock_timeout. Note After a lock timeout any transaction containing the statement, is rolled back or canceled by SQL Server 2000 (bug#352640 was filed). This behavior is different from that of SQL Server 7.0. With SQL Server 7.0, the application must have an error handler that can trap error 1222 and if an application does not trap the error, it can proceed unaware that an individual statement within a transaction has been canceled, and errors can occur because statements later in the transaction may depend on the statement that was never executed. Bug#352640 is fixed in hotfix build 8.00.266 whereby a lock timeout will only Internal Lock Timeout At time, internal operations within SQL Server will attempt to acquire locks via lock manager. Typically, these lock requests are issued with “no waiting.” For example, the ghost record processing might try to clean up rows on a particular page, and before it can do that, it needs to lock the page. Thus, the ghost record manager will request a page lock with no wait so that if it cannot lock the page, it will just move on to other pages; it can always come back to this page later. If you look at SQL Profiler Lock: Timeout events, internal lock timeout typically have a duration value of zero. Lock Duration Lock Mode and Transaction Isolation Level For REPEATABLE READ transaction isolation level, update locks are held until data is read and processed, unless promoted to exclusive locks. "Data is processed" means that we have decided whether the row in question matched the search criteria; if not then the update lock is released, otherwise, we get an exclusive lock and make the modification. Consider the following query: use northwind go dbcc traceon(3604, 1200, 1211) -- turn on lock tracing -- and disable escalation go set transaction isolation level repeatable read begin tran update dbo.[order details] set discount = convert (real, discount) where discount = 0.0 exec sp_lock Update locks are promoted to exclusive locks when there is a match; otherwise, the update lock is released. The sp_lock output verifies that the SPID does not hold any update locks or shared locks at the end of the query. Lock escalation is turned off so that exclusive table lock is not held at the end. Warning Do not use trace flag 1200 in a production environment because it produces a lot of output and slows down the server. Trace flag 1211 should not be used unless you have done extensive study to make sure it helps with performance. These trace flags are used here for illustration and learning purposes only. Lock Ownership Most of the locking discussion in this lesson relates to locks owned by “transactions.” In addition to transaction, cursor and session can be owners of locks and they both affect how long locks are held. For every row that is fetched, when SCROLL_LOCKS option is used, regardless of the state of a transaction, a cursor lock is held until the next row is fetched or when the cursor is closed. Locks owned by session are outside the scope of a transaction. The duration of these locks are bounded by the connection and the process will continue to hold these locks until the process disconnects. A typical lock owned by session is the database (DB) lock. Locking – Read Committed Scan Under read committed isolation level, when database pages are scanned, shared locks are held when the page is read and processed. The shared locks are released “behind” the scan and allow other transactions to update rows. It is important to note that the shared lock currently acquired will not be released until shared lock for the next page is successfully acquired (this is commonly know as “crabbing”). If the same pages are scanned again, rows may be modified or deleted by other transactions. Locking – Repeatable Read Scan Under repeatable read isolation level, when database pages are scanned, shared locks are held when the page is read and processed. SQL Server continues to hold these shared locks, thus preventing other transactions to update rows. If the same pages are scanned again, previously scanned rows will not change but new rows may be added by other transactions. Locking – Serializable Read Scan Under serializable read isolation level, when database pages are scanned, shared locks are held not only on rows but also on scanned key range. SQL Server continues to hold these shared locks until the end of transaction. Because key range locks are held, not only will this prevent other transactions from modifying the rows, no new rows can be inserted. Prefetch and Isolation Level Prefetch and Locking Behavior The prefetch feature is available for use with SQL Server 7.0 and SQL Server 2000. When searching for data using a nonclustered index, the index is searched for a particular value. When that value is found, the index points to the disk address. The traditional approach would be to immediately issue an I/O for that row, given the disk address. The result is one synchronous I/O per row and, at most, one disk at a time working to evaluate the query. This does not take advantage of striped disk sets. The prefetch feature takes a different approach. It continues looking for more record pointers in the nonclustered index. When it has collected a number of them, it provides the storage engine with prefetch hints. These hints tell the storage engine that the query processor will need these particular records soon. The storage engine can now issue several I/Os simultaneously, taking advantage of striped disk sets to execute multiple operations simultaneously. For example, if the engine is scanning a nonclustered index to determine which rows qualify but will eventually need to visit the data page as well to access columns that are not in the index, it may decide to submit asynchronous page read requests for a group of qualifying rows. The prefetch data pages are then revisited later to avoid waiting for each individual page read to complete in a serial fashion. This data access path requires that a lock be held between the prefetch request and the row lookup to stabilize the row on the page so it is not to be moved by a page split or clustered key update. For our example, the isolation level of the query is escalated to REPEATABLE READ, overriding the transaction isolation level. With SQL Server 7.0 and SQL Server 2000, portions of a transaction can execute at a different transaction isolation level than the entire transaction itself. This is implemented as lock classes. Lock classes are used to control lock lifetime when portions of a transaction need to execute at a stricter isolation level than the underlying transaction. Unfortunately, in SQL Server 7.0 and SQL Server 2000, the lock class is created at the topmost operator of the query and hence released only at the end of the query. Currently there is no support to release the lock (lock class) after the row has been discarded or fetched by the filter or join operator. This is because isolation level can be set at the query level via a lock class, but no lower. Because of this, locks acquired during the query will not be released until the query completes. If prefetch is occurring you may see a single SPID that holds hundreds of Shared KEY or PAG locks even though the connection’s isolation level is READ COMMITTED. Isolation level can be determined from DBCC PSS output. For details about this behavior see “SOX001109700040 INF: Queries with PREFETCH in the plan hold lock until the end of transaction”. Other Locking Mechanism Lock manager does not manage latches and spinlocks. Latches Latches are internal mechanisms used to protect pages while doing operations such as placing a row physically on a page, compressing space on a page, or retrieving rows from a page. Latches can roughly be divided into I/O latches and non-I/O latches. If you see a high number of non-I/O related latches, SQL Server is usually doing a large number of hash or sort operations in tempdb. You can monitor latch activities via DBCC SQLPERF(‘WAITSTATS’) command. Spinlock A spinlock is an internal data structure that is used to protect vital information that is shared within SQL Server. On a multi-processor machine, when SQL Server tries to access a particular resource protected by a spinlock, it must first acquire the spinlock. If it fails, it executes a loop that will check to see if the lock is available and if not, decrements a counter. If the counter reaches zero, it yields the processor to another thread and goes into a “sleep” (wait) state for a pre-determined amount of time. When it wakes, hopefully, the lock is free and available. If not, the loop starts again and it is terminated only when the lock is acquired. The reason for implementing a spinlock is that it is probably less costly to “spin” for a short time rather than yielding the processor. Yielding the processor will force an expensive context switch where:  The old thread’s state must be saved  The new thread’s state must be reloaded  The data stored in the L1 and L2 cache are useless to the processor On a single-processor computer, the loop is not useful because no other thread can be running and thus, no one can release the spinlock for the currently executing thread to acquire. In this situation, the thread yields the processor immediately. Lesson 2: Concepts – Batch and Transaction This lesson outlines some of the common causes that contribute to the perception of a slow server. What You Will Learn After completing this lesson, you will be able to:  Review batch processing and error checking.  Review explicit, implicit and autocommit transactions and transaction nesting level.  Discuss how commit and rollback transaction done in stored procedure and trigger affects transaction nesting level.  Discuss various transaction isolation level and their impact on locking.  Discuss the difference between aborting a statement, a transaction, and a batch.  Describe how @@error, @@transcount, and @@rowcount can be used for error checking and handling. Recommended Reading  Charter 12 “Transactions and Triggers”, Inside SQL Server 2000 by Kalen Delaney Batch Definition SQL Profiler Statements and Batches To help further your understanding of what is a batch and what is a statement, you can use SQL Profiler to study the definition of batch and statement.  Try This: Using SQL Profiler to Analyze Batch 1. Log on to a server with Query Analyzer 2. Startup the SQL Profiler against the same server 3. Start a trace using the “StandardSQLProfiler” template 4. Execute the following using Query Analyzer: SELECT @@VERSION SELECT @@SPID The ‘SQL:BatchCompleted’ event is captured by the trace. It shows both the statements as a single batch. 5. Now execute the following using Query Analyzer {call sp_who()} What shows up? The ‘RPC:Completed’ with the sp_who information. RPC is simply another entry point to the SQL Server to call stored procedures with native data types. This allows one to avoid parsing. The ‘RPC:Completed’ event should be considered the same as a batch for the purposes of this discussion. Stop the current trace and start a new trace using the “SQLProfilerTSQL_SPs” template. Issue the same command as outlines in step 5 above. Looking at the output, not only can you see the batch markers but each statement as executed within the batch. Autocommit, Explicit, and Implicit Transaction Autocommit Transaction Mode (Default) Autocommit mode is the default transaction management mode of SQL Server. Every Transact-SQL statement, whether it is a standalone statement or part of a batch, is committed or rolled back when it completes. If a statement completes successfully, it is committed; if it encounters any error, it is rolled back. A SQL Server connection operates in autocommit mode whenever this default mode has not been overridden by either explicit or implicit transactions. Autocommit mode is also the default mode for ADO, OLE DB, ODBC, and DB-Library. A SQL Server connection operates in autocommit mode until a BEGIN TRANSACTION statement starts an explicit transaction, or implicit transaction mode is set on. When the explicit transaction is committed or rolled back, or when implicit transaction mode is turned off, SQL Server returns to autocommit mode. Explicit Transaction Mode An explicit transaction is a transaction that starts with a BEGIN TRANSACTION statement. An explicit transaction can contain one or more statements and must be terminated by either a COMMIT TRANSACTION or a ROLLBACK TRANSACTION statement. Implicit Transaction Mode SQL Server can automatically or, more precisely, implicitly start a transaction for you if a SET IMPLICIT_TRANSACTIONS ON statement is run or if the implicit transaction option is turned on globally by running sp_configure ‘user options’ 2. (Actually, the bit mask 0x2 must be turned on for the user option so you might have to perform an ‘OR’ operation with the existing user option value.) See SQL Server 2000 Books Online on how to turn on implicit transaction under ODBC and OLE DB (acdata.chm::/ac_8_md_06_2g6r.htm). Transaction Nesting Explicit transactions can be nested. Committing inner transactions is ignored by SQL Server other than to decrements @@TRANCOUNT. The transaction is either committed or rolled back based on the action taken at the end of the outermost transaction. If the outer transaction is committed, the inner nested transactions are also committed. If the outer transaction is rolled back, then all inner transactions are also rolled back, regardless of whether the inner transactions were individually committed. Each call to COMMIT TRANSACTION applies to the last executed BEGIN TRANSACTION. If the BEGIN TRANSACTION statements are nested, then a COMMIT statement applies only to the last nested transaction, which is the innermost transaction. Even if a COMMIT TRANSACTION transaction_name statement within a nested transaction refers to the transaction name of the outer transaction, the commit applies only to the innermost transaction. If a ROLLBACK TRANSACTION statement without a transaction_name parameter is executed at any level of a set of nested transaction, it rolls back all the nested transactions, including the outermost transaction. The @@TRANCOUNT function records the current transaction nesting level. Each BEGIN TRANSACTION statement increments @@TRANCOUNT by one. Each COMMIT TRANSACTION statement decrements @@TRANCOUNT by one. A ROLLBACK TRANSACTION statement that does not have a transaction name rolls back all nested transactions and decrements @@TRANCOUNT to 0. A ROLLBACK TRANSACTION that uses the transaction name of the outermost transaction in a set of nested transactions rolls back all the nested transactions and decrements @@TRANCOUNT to 0. When you are unsure if you are already in a transaction, SELECT @@TRANCOUNT to determine whether it is 1 or more. If @@TRANCOUNT is 0 you are not in a transaction. You can also find the transaction nesting level by checking the sysprocess.open_tran column. See SQL Server 2000 Books Online topic “Nesting Transactions” (acdata.chm::/ac_8_md_06_66nq.htm) for more information. Statement, Transaction, and Batch Abort One batch can have many statements and one transaction can have multiple statements, also. One transaction can span multiple batches and one batch can have multiple transactions. Statement Abort Currently executing statement is aborted. This can be a bit confusing when you start talking about statements in a trigger or stored procedure. Let us look closely at the following trigger: CREATE TRIGGER TRG8134 ON TBL8134 AFTER INSERT AS BEGIN SELECT 1/0 SELECT 'Next command in trigger' END To fire the INSERT trigger, the batch could be as simple as ‘INSERT INTO TBL8134 VALUES(1)’. However, the trigger contains two statements that must be executed as part of the batch to satisfy the clients insert request. When the ‘SELECT 1/0’ causes the divide by zero error, a statement abort is issued for the ‘SELECT 1/0’ statement. Batch and Transaction Abort On SQL Server 2000 (and SQL Server 7.0) whenever a non-informational error is encountered in a trigger, the statement abort is promoted to a batch and transactional abort. Thus, in the example the statement abort for ‘select 1/0’ promotion results in an entire batch abort. No further statements in the trigger or batch will be executed and a rollback is issued. On SQL Server 6.5, the statement aborts immediately and results in a transaction abort. However, the rest of the statements within the trigger are executed. This trigger could return ‘Next command in trigger’ as a result set. Once the trigger completes the batch abort promotion takes effect. Conversely, submitting a similar set of statements in a standalone batch can result in different behavior. SELECT 1/0 SELECT 'Next command in batch' Not considering the set option possibilities, a divide by zero error generally results in a statement abort. Since it is not in a trigger, the promotion to a batch abort is avoided and subsequent SELECT statement can execute. The programmer should add an “if @@ERROR” check immediately after the ‘select 1/0’ to T-SQL execution to control the flow correctly. Aborting and Set Options ARITHABORT If SET ARITHABORT is ON, these error conditions cause the query or batch to terminate. If the errors occur in a transaction, the transaction is rolled back. If SET ARITHABORT is OFF and one of these errors occurs, a warning message is displayed, and NULL is assigned to the result of the arithmetic operation. When an INSERT, DELETE, or UPDATE statement encounters an arithmetic error (overflow, divide-by-zero, or a domain error) during expression evaluation when SET ARITHABORT is OFF, SQL Server inserts or updates a NULL value. If the target column is not nullable, the insert or update action fails and the user receives an error. XACT_ABORT When SET XACT_ABORT is ON, if a Transact-SQL statement raises a run-time error, the entire transaction is terminated and rolled back. When OFF, only the Transact-SQL statement that raised the error is rolled back and the transaction continues processing. Compile errors, such as syntax errors, are not affected by SET XACT_ABORT. For example: CREATE TABLE t1 (a int PRIMARY KEY) CREATE TABLE t2 (a int REFERENCES t1(a)) GO INSERT INTO t1 VALUES (1) INSERT INTO t1 VALUES (3) INSERT INTO t1 VALUES (4) INSERT INTO t1 VALUES (6) GO SET XACT_ABORT OFF GO BEGIN TRAN INSERT INTO t2 VALUES (1) INSERT INTO t2 VALUES (2) /* Foreign key error */ INSERT INTO t2 VALUES (3) COMMIT TRAN SELECT 'Continue running batch 1...' GO SET XACT_ABORT ON GO BEGIN TRAN INSERT INTO t2 VALUES (4) INSERT INTO t2 VALUES (5) /* Foreign key error */ INSERT INTO t2 VALUES (6) COMMIT TRAN SELECT 'Continue running batch 2...' GO /* Select shows only keys 1 and 3 added. Key 2 insert failed and was rolled back, but XACT_ABORT was OFF and rest of transaction succeeded. Key 5 insert error with XACT_ABORT ON caused all of the second transaction to roll back. Also note that 'Continue running batch 2...' is not Returned to indicate that the batch is aborted. */ SELECT * FROM t2 GO DROP TABLE t2 DROP TABLE t1 GO Compile and Run-time Errors Compile Errors Compile errors are encountered during syntax checks, security checks, and other general operations to prepare the batch for execution. These errors can prevent the optimization of the query and thus lead to immediate abort. The statement is not run and the batch is aborted. The transaction state is generally left untouched. For example, assume there are four statements in a particular batch. If the third statement has a syntax error, none of the statements in the batch is executed. Optimization Errors Optimization errors would include rare situations where the statement encounters a problem when attempting to build an optimal execution plan. Example: “too many tables referenced in the query” error is reported because a “work table” was added to the plan. Runtime Errors Runtime errors are those that are encountered during the execution of the query. Consider the following batch: SELECT * FROM pubs.dbo.titles UPDATE pubs.dbo.authors SET au_lname = au_lname SELECT * FROM foo UPDATE pubs.dbo.authors SET au_lname = au_lname If you run the above statements in a batch, the first two statements will be executed, the third statement will fail because table foo does not exist, and the batch will terminate. Deferred Name Resolution is the feature that allows this batch to start executing before resolving the object foo. This feature allows SQL Server to delay object resolution and place a “placeholder” in the query’s execution. The object referenced by the placeholder is resolved until the query is executed. In our example, the execution of the statement “SELECT * FROM foo” will trigger another compile process to resolve the name again. This time, error message 208 is returned. Error: 208, Level 16, State 1, Line 1 Invalid object name 'foo'. Message 208 can be encountered as a runtime or compile error depending on whether the Deferred Name Resolution feature is available. In SQL Server 6.5 this would be considered a compile error and on SQL Server 2000 (and SQL Server7.0) as a runtime error due to Deferred Name Resolution. In the following example, if a trigger referenced authors2, the error is detected as SQL Server attempts to execute the trigger. However, under SQL Server 6.5 the create trigger statement fails because authors2 does not exist at compile time. When errors are encountered in a trigger, generally, the statement, batch, and transaction are aborted. You should be able to observe this by running the following script in pubs database: Create table tblTest(iID int) go create trigger trgInsert on tblTest for INSERT as begin select * from authors select * from authors2 select * from titles end go begin tran select 'Before' insert into tblTest values(1) select 'After' go select @@TRANCOUNT go When run in a batch, the statement and the batch are aborted but the transaction remains active. The follow script illustrates this: begin tran select 'Before' select * from authors2 select 'After' go select @@TRANCOUNT go One other factor in a compile versus runtime error is implicit data type conversions. If you were to run the following statements on SQL Server 6.5 and SQL Server 2000 (and SQL Server 7.0): create table tblData(dtData datetime) go select 1 insert into tblData values(12/13/99) go On SQL Server 6.5, you get an error before execution of the batch begins so no statements are executed and the batch is aborted. Error: 206, Level 16, State 2, Line 2 Operand type clash: int is incompatible with datetime On SQL Server 2000, you get the default value (1900-01-01 00:00:00.000) inserted into the table. SQL Server 2000 implicit data type conversion treats this as integer division. The integer division of 12/13/99 is 0, so the default date and time value is inserted, no error returned. To correct the problem on either version is to wrap the date string with quotes. See Bug #56118 (sqlbug_70) for more details about this situation. Another example of a runtime error is a 605 message. Error: 605 Attempt to fetch logical page %S_PGID in database '%.*ls' belongs to object '%.*ls', not to object '%.*ls'. A 605 error is always a runtime error. However, depending on the transaction isolation level, (e.g. using the NOLOCK lock hint), established by the SPID the handling of the error can vary. Specifically, a 605 error is considered an ACCESS error. Errors associated with buffer and page access are found in the 600 series of errors. When the error is encountered, the isolation level of the SPID is examined to determine proper handling based on information or fatal error level. Transaction Error Checking Not all errors cause transactions to automatically rollback. Although it is difficult to determine exactly which errors will rollback transactions and which errors will not, the main idea here is that programmers must perform error checking and handle errors appropriately. Error Handling Raiserror Details Raiserror seems to be a source of confusion but is really rather simple. Raiserror with severity levels of 20 or higher will terminate the connection. Of course, when the connection is terminated a full rollback of any open transaction will immediately be instantiated by the SQL Server (except distributed transaction with DTC involved). Severity levels lower than 20 will simply result in the error message being returned to the client. They do not affect the transaction scope of the connection. Consider the following batch: use pubs begin tran update authors set au_lname = 'smith' raiserror ('This is bad', 19, 1) with log select @@trancount With severity set at 19, the 'select @@trancount' will be executed after the raiserror statement and will return a value of 1. If severity is changed to 20, then the select statement will not run and the connection is broken. Important Error handling must occur not only in T-SQL batches and stored procedures, but also in application program code. Transactions and Triggers (1 of 2) Basic behavior assumes the implicit transactions setting is set to OFF. This behavior makes it possible to identify business logic errors in a trigger, raise an error, rollback the action, and add an audit table entry. Logically, the insert to the audit table cannot take place before the ROLLBACK action and you would not want to build in the audit table insert into every applications error handler that violated the business rule of the trigger. For more information, see also… SQL Server 2000 Books Online topic “Rollbacks in stored procedure and triggers“ (acdata.chm::/ac_8_md_06_4qcz.htm) IMPLICIT_TRANSACTIONS ON Behavior The behavior of firing other triggers on the same table can be tricky. Say you added a trigger that checks the CODE field. Read only versions of the rows contain the code ‘RO’ and read/write versions use ‘RW.’ Whenever someone tries to delete a row with a code ‘RO’ the trigger issues the rollback and logs an audit table entry. However, you also have a second trigger that is responsible for cascading delete operations. One client could issue the delete without implicit transactions on and only the current trigger would execute and then terminate the batch. However, a second client with implicit transactions on could issue the same delete and the secondary trigger would fire. You end up with a situation in which the cascading delete operations can take place (are committed) but the initial row remains in the table because of the rollback operation. None of the delete operations should be allowed but because the transaction scope was restarted because of the implicit transactions setting, they did. Transactions and Triggers (2 of 2) It is extremely difficult to determine the execution state of a trigger when using explicit rollback statements in combination with implicit transactions. The RETURN statement is not allowed to return a value. The only way I have found to set the @@ERROR is using a ‘raiserror’ as the last execution statement in the last trigger to execute. If you modify the example, this following RAISERROR statement will set @@ERROR to 50000: CREATE TRIGGER trgTest on tblTest for INSERT AS BEGIN ROLLBACK INSERT INTO tblAudit VALUES (1) RAISERROR('This is bad', 14,1) END However, this value does not carry over to a secondary trigger for the same table. If you raise an error at the end of the first trigger and then look at @@ERROR in the secondary trigger the @@ERROR remains 0. Carrying Forward an Active/Open Transaction It is possible to exit from a trigger and carry forward an open transaction by issuing a BEGIN TRAN or by setting implicit transaction on and doing INSERT, UPDATE, or DELETE. Warning It is never recommended that a trigger call BEGIN TRANSACTION. By doing this you increment the transaction count. Invalid code logic, not calling commit transaction, can lead to a situation where the transaction count remains elevated upon exit of the trigger. Transaction Count The behavior is better explained by understanding how the server works. It does not matter whether you are in a transaction, when a modification takes place the transaction count is incremented. So, in the simplest form, during the processing of an insert the transaction count is 1. On completion of the insert, the server will commit (and thus decrement the transaction count). If the commit identifies the transaction count has returned to 0, the actual commit processing is completed. Issuing a commit when the transaction count is greater than 1 simply decrements the nested transaction counter. Thus, when we enter a trigger, the transaction count is 1. At the completion of the trigger, the transaction count will be 0 due to the commit issued at the end of the modification statement (insert). In our example, if the connection was already in a transaction and called the second INSERT, since implicit transaction is ON, the transaction count in the trigger will be 2 as long as the ROLLBACK is not executed. At the end of the insert, the commit is again issued to decrement the transaction reference count to 1. However, the value does not return to 0 so the transaction remains open/active. Subsequent triggers are only fired if the transaction count at the end of the trigger remains greater than or equal to 1. The key to continuation of secondary triggers and the batch is the transaction count at the end of a trigger execution. If the trigger that performs a rollback has done an explicit begin transaction or uses implicit transactions, subsequent triggers and the batch will continue. If the transaction count is not 1 or greater, subsequent triggers and the batch will not execute. Warning Forcing the transaction count after issuing a rollback is dangerous because you can easily loose track of your transaction nesting level. When performing an explicit rollback in a trigger, you should immediately issue a return statement to maintain consistent behavior between a connection with and without implicit transaction settings. This will force the trigger(s) and batch to terminate immediately. One of the methods of dealing with this issue is to run ‘SET IMPLICIT_TRANSACTIONS OFF’ as the first statement of any trigger. Other methods may entails checking @@TRANCOUNT at the end of the trigger and continue to COMMIT the transaction as long as @@TRANCOUNT is greater than 1. Examples The following examples are based on this table: create table tbl50000Insert (iID int NOT NULL) go Note If more than one trigger is used, to guarantee the trigger firing sequence, the sp_settriggerorder command should be used. This command is omitted in these examples to simplify the complexity of the statements. First Example In the first example, the second trigger was never fired and the batch, starting with the insert statement, was aborted. Thus, the print statement was never issued. print('Trigger issues rollback - cancels batch') go create trigger trg50000Insert on tbl50000Insert for INSERT as begin select 'Inserted', * from inserted rollback tran select 'End of trigger', @@TRANCOUNT as 'TRANCOUNT' end go create trigger trg50000Insert2 on tbl50000Insert for INSERT as begin select 'In Trigger2' select 'Trigger 2 Inserted', * from inserted end go insert into tbl50000Insert values(1) print('---------------------- In same batch') select * from tbl50000Insert go -- Cleanup drop trigger trg50000Insert drop trigger trg50000Insert2 go delete from tbl50000Insert Second Example The next example shows that since a new transaction is started, the second trigger will be fired and the print statement in the batch will be executed. Note that the insert is rolled back. print('Trigger issues rollback - increases tran count to continue batch') go create trigger trg50000Insert on tbl50000Insert for INSERT as begin select 'Inserted', * from inserted rollback tran begin tran end go create trigger trg50000Insert2 on tbl50000Insert for INSERT as begin select 'In Trigger2' select 'Trigger 2 Inserted', * from inserted end go insert into tbl50000Insert values(2) print('---------------------- In same batch') select * from tbl50000Insert go -- Cleanup drop trigger trg50000Insert drop trigger trg50000Insert2 go delete from tbl50000Insert Third Example In the third example, the raiserror statement is used to set the @@ERROR value and the BEGIN TRAN statement is used in the trigger to allow the batch to continue to run. print('Trigger issues rollback - uses raiserror to set @@ERROR') go create trigger trg50000Insert on tbl50000Insert for INSERT as begin select 'Inserted', * from inserted rollback tran begin tran -- Increase @@trancount to allow -- batch to continue select @@trancount as ‘Trancount’ raiserror('This is from the trigger', 14,1) end go insert into tbl50000Insert values(3) select @@ERROR as 'ERROR', @@TRANCOUNT as 'Trancount' go -- Cleanup drop trigger trg50000Insert go delete from tbl50000Insert Fourth Example For the fourth example, a second trigger is added to illustrate the fact that @@ERROR value set in the first trigger will not be seen in the second trigger nor will it show up in the batch after the second trigger is fired. print('Trigger issues rollback - uses raiserror to set @@ERROR, not seen in second trigger and cleared in batch') go create trigger trg50000Insert on tbl50000Insert for INSERT as begin select 'Inserted', * from inserted rollback begin tran -- Increase @@trancount to -- allow batch to continue select @@TRANCOUNT as 'Trancount' raiserror('This is from the trigger', 14,1) end go create trigger trg50000Insert2 on tbl50000Insert for INSERT as begin select @@ERROR as 'ERROR', @@TRANCOUNT as 'Trancount' end go insert into tbl50000Insert values(4) select @@ERROR as 'ERROR', @@TRANCOUNT as 'Trancount' go -- Cleanup drop trigger trg50000Insert drop trigger trg50000Insert2 go delete from tbl50000Insert Lesson 3: Concepts – Locks and Applications This lesson outlines some of the common causes that contribute to the perception of a slow server. What You Will Learn After completing this lesson, you will be able to:  Explain how lock hints are used and their impact.  Discuss the effect on locking when an application uses Microsoft Transaction Server.  Identify the different kinds of deadlocks including distributed deadlock. Recommended Reading  Charter 14 “Locking”, Inside SQL Server 2000 by Kalen Delaney  Charter 16 “Query Tuning”, Inside SQL Server 2000 by Kalen Delaney Q239753 – Deadlock Situation Not Detected by SQL Server Q288752 – Blocked SPID Not Participating in Deadlock May Incorrectly be Chosen as victim Locking Hints UPDLOCK If update locks are used instead of shared locks while reading a table, the locks are held until the end of the statement or transaction. UPDLOCK has the advantage of allowing you to read data (without blocking other readers) and update it later with the assurance that the data has not changed since you last read it. READPAST READPAST is an optimizer hint for use with SELECT statements. When this hint is used, SQL Server will read past locked rows. For example, assume table T1 contains a single integer column with the values of 1, 2, 3, 4, and 5. If transaction A changes the value of 3 to 8 but has not yet committed, a SELECT * FROM T1 (READPAST) yields values 1, 2, 4, 5. Tip READPAST only applies to transactions operating at READ COMMITTED isolation and only reads past row-level locks. This lock hint can be used to implement a work queue on a SQL Server table. For example, assume there are many external work requests being thrown into a table and they should be serviced in approximate insertion order but they do not have to be completely FIFO. If you have 4 worker threads consuming work items from the queue they could each pick up a record using read past locking and then delete the entry from the queue and commit when they're done. If they fail, they could rollback, leaving the entry on the queue for the next worker thread to pick up. Caution The READPAST hint is not compatible with HOLDLOCK.  Try This: Using Locking Hints 1. Open a Query Window and connect to the pubs database. 2. Execute the following statements (--Conn 1 is optional to help you keep track of each connection): BEGIN TRANSACTION -- Conn 1 UPDATE titles SET price = price * 0.9 WHERE title_id = 'BU1032' 3. Open a second connection and execute the following statements: SELECT @@lock_timeout -- Conn 2 GO SELECT * FROM titles SELECT * FROM authors 4. Open a third connection and execute the following statements: SET LOCK_TIMEOUT 0 -- Conn 3 SELECT * FROM titles SELECT * FROM authors 5. Open a fourth connection and execute the following statement: SELECT * FROM titles (READPAST) -- Conn 4 WHERE title_ID < 'C' SELECT * FROM authors How many records were returned? 3 6. Open a fifth connection and execute the following statement: SELECT * FROM titles (NOLOCK) -- Conn 5 WHERE title_ID 0 the lock manager also checks for deadlocks every time a SPID gets blocked. So a single deadlock will trigger 20 seconds of more immediate deadlock detection, but if no additional deadlocks occur in that 20 seconds, the lock manager no longer checks for deadlocks at each block and detection again only happens every 5 seconds. Although normally not needed, you may use trace flag -T1205 to trace the deadlock detection process. Note Please note the distinction between application lock and other locks’ deadlock detection. For application lock, we do not rollback the transaction of the deadlock victim but simply return a -3 to sp_getapplock, which the application needs to handle itself. Deadlock Resolution How is a deadlock resolved? SQL Server picks one of the connections as a deadlock victim. The victim is chosen based on either which is the least expensive transaction (calculated using the number and size of the log records) to roll back or in which process “SET DEADLOCK_PRIORITY LOW” is specified. The victim’s transaction is rolled back, held locks are released, and SQL Server sends error 1205 to the victim’s client application to notify it that it was chosen as a victim. The other process can then obtain access to the resource it was waiting on and continue. Error 1205: Your transaction (process ID #%d) was deadlocked with another process and has been chosen as the deadlock victim. Rerun your transaction. Symptoms of deadlocking Error 1205 usually is not written to the SQL Server errorlog. Unfortunately, you cannot use sp_altermessage to cause 1205 to be written to the errorlog. If the client application does not capture and display error 1205, some of the symptoms of deadlock occurring are:  Clients complain of mysteriously canceled queries when using certain features of an application.  May be accompanied by excessive blocking. Lock contention increases the chances that a deadlock will occur. Triggers and Deadlock Triggers promote the deadlock priority of the SPID for the life of the trigger execution when the DEADLOCK PRIORITY is not set to low. When a statement in a trigger causes a deadlock to occur, the SPID executing the trigger is given preferential treatment and will not become the victim. Warning Bug 235794 is filed against SQL Server 2000 where a blocked SPID that is not a participant of a deadlock may incorrectly be chosen as a deadlock victim if the SPID is blocked by one of the deadlock participants and the SPID has the least amount of transaction logging. See KB article Q288752: “Blocked Spid Not Participating in Deadlock May Incorrectly be Chosen as victim” for more information. Distributed Deadlock – Scenario 1 Distributed Deadlocks The term distributed deadlock is ambiguous. There are many types of distributed deadlocks. Scenario 1 Client application opens connection A, begins a transaction, acquires some locks, opens connection B, connection B gets blocked by A but the application is designed to not commit A’s transaction until B completes. Note SQL Server has no way of knowing that connection A is somehow dependent on B – they are two distinct connections with two distinct transactions. This situation is discussed in scenario #4 in “Q224453 INF: Understanding and Resolving SQL Server 7.0 Blocking Problems”. Distributed Deadlock – Scenario 2 Scenario 2 Distributed deadlock involving bound connections. Two connections can be bound into a single transaction context with sp_getbindtoken/sp_bindsession or via DTC. Spid 60 enlists in a transaction with spid 61. A third spid 62 is blocked by spid 60, but spid 61 is blocked by spid 62. Because they are doing work in the same transaction, spid 60 cannot commit until spid 61 finishes his work, but spid 61 is blocked by 62 who is blocked by 60. This scenario is described in article “Q239753 - Deadlock Situation Not Detected by SQL Server.” Note SQL Server 6.5 and 7.0 do not detect this deadlock. The SQL Server 2000 deadlock detection algorithm has been enhanced to detect this type of distributed deadlock. The diagram in the slide illustrates this situation. Resources locked by a spid are below that spid (in a box). Arrows indicate blocking and are drawn from the blocked spid to the resource that the spid requires. A circle represents a transaction; spids in the same transaction are shown in the same circle. Distributed Deadlock – Scenario 3 Scenario 3 Distributed deadlock involving linked servers or server-to-server RPC. Spid 60 on Server 1 executes a stored procedure on Server 2 via linked server. This stored procedure does a loopback linked server query against a table on Server 1, and this connection is blocked by a lock held by Spid 60. Note No version of SQL Server is currently designed to detect this distributed deadlock. Lesson 4: Information Collection and Analysis This lesson outlines some of the common causes that contribute to the perception of a slow server. What You Will Learn After completing this lesson, you will be able to:  Identify specific information needed for troubleshooting issues.  Locate and collect information needed for troubleshooting issues.  Analyze output of DBCC Inputbuffer, DBCC PSS, and DBCC Page commands.  Review information collected from master.dbo.sysprocesses table.  Review information collected from master.dbo.syslockinfo table.  Review output of sp_who, sp_who2, sp_lock.  Analyze Profiler log for query usage pattern.  Review output of trace flags to help troubleshoot deadlocks. Recommended Reading Q244455 - INF: Definition of Sysprocesses Waittype and Lastwaittype Fields Q244456 - INF: Description of DBCC PSS Command for SQL Server 7.0 Q271509 - INF: How to Monitor SQL Server 2000 Blocking Q251004 - How to Monitor SQL Server 7.0 Blocking Q224453 - Understanding and Resolving SQL Server 7.0 Blocking Problem Q282749 – BUG: Deadlock information reported with SQL Server 2000 Profiler Locking and Blocking  Try This: Examine Blocked Processes 1. Open a Query Window and connect to the pubs database. Execute the following statements: BEGIN TRAN -- connection 1 UPDATE titles SET price = price + 1 2. Open another connection and execute the following statement: SELECT * FROM titles-- connection 2 3. Open a third connection and execute sp_who; note the process id (spid) of the blocked process. (Connection 3) 4. In the same connection, execute the following: SELECT spid, cmd, waittype FROM master..sysprocesses WHERE waittype 0 -- connection 3 5. Do not close any of the connections! What was the wait type of the blocked process?  Try This: Look at locks held Assumes all your connections are still open from the previous exercise. • Execute sp_lock -- Connection 3 What locks is the process from the previous example holding? Make sure you run ROLLBACK TRAN in Connection 1 to clean up your transaction. Collecting Information See Module 2 for more about how to gather this information using various tools. Recognizing Blocking Problems How to Recognize Blocking Problems  Users complain about poor performance at a certain time of day, or after a certain number of users connect.  SELECT * FROM sysprocesses or sp_who2 shows non-zero values in the blocked or BlkBy column.  More severe blocking incidents will have long blocking chains or large sysprocesses.waittime values for blocked spids.  Possibl
Table of Contents Header Files The #define Guard Header File Dependencies Inline Functions The -inl.h Files Function Parameter Ordering Names and Order of Includes Scoping Namespaces Nested Classes Nonmember, Static Member, and Global Functions Local Variables Static and Global Variables Classes Doing Work in Constructors Default Constructors Explicit Constructors Copy Constructors Structs vs. Classes Inheritance Multiple Inheritance Interfaces Operator Overloading Access Control Declaration Order Write Short Functions Google-Specific Magic Smart Pointers cpplint Other C++ Features Reference Arguments Function Overloading Default Arguments Variable-Length Arrays and alloca() Friends Exceptions Run-Time Type Information (RTTI) Casting Streams Preincrement and Predecrement Use of const Integer Types 64-bit Portability Preprocessor Macros 0 and NULL sizeof Boost C++0x Naming General Naming Rules File Names Type Names Variable Names Constant Names Function Names Namespace Names Enumerator Names Macro Names Exceptions to Naming Rules Comments Comment Style File Comments Class Comments Function Comments Variable Comments Implementation Comments Punctuation, Spelling and Grammar TODO Comments Deprecation Comments Formatting Line Length Non-ASCII Characters Spaces vs. Tabs Function Declarations and Definitions Function Calls Conditionals Loops and Switch Statements Pointer and Reference Expressions Boolean Expressions Return Values Variable and Array Initialization Preprocessor Directives Class Format Constructor Initializer Lists Namespace Formatting Horizontal Whitespace Vertical Whitespace Exceptions to the Rules Existing Non-conformant Code Windows Code Important Note Displaying Hidden Details in this Guide link ▶This style guide contains many details that are initially hidden from view. They are marked by the triangle icon, which you see here on your left. Click it now. You should see "Hooray" appear below. Hooray! Now you know you can expand points to get more details. Alternatively, there's an "expand all" at the top of this document. Background C++ is the main development language used by many of Google's open-source projects. As every C++ programmer knows, the language has many powerful features, but this power brings with it complexity, which in turn can make code more bug-prone and harder to read and maintain. The goal of this guide is to manage this complexity by describing in detail the dos and don'ts of writing C++ code. These rules exist to keep the code base manageable while still allowing coders to use C++ language features productively. Style, also known as readability, is what we call the conventions that govern our C++ code. The term Style is a bit of a misnomer, since these conventions cover far more than just source file formatting. One way in which we keep the code base manageable is by enforcing consistency. It is very important that any programmer be able to look at another's code and quickly understand it. Maintaining a uniform style and following conventions means that we can more easily use "pattern-matching" to infer what various symbols are and what invariants are true about them. Creating common, required idioms and patterns makes code much easier to understand. In some cases there might be good arguments for changing certain style rules, but we nonetheless keep things as they are in order to preserve consistency. Another issue this guide addresses is that of C++ feature bloat. C++ is a huge language with many advanced features. In some cases we constrain, or even ban, use of certain features. We do this to keep code simple and to avoid the various common errors and problems that these features can cause. This guide lists these features and explains why their use is restricted. Open-source projects developed by Google conform to the requirements in this guide. Note that this guide is not a C++ tutorial: we assume that the reader is familiar with the language. Header Files In general, every .cc file should have an associated .h file. There are some common exceptions, such as unittests and small .cc files containing just a main() function. Correct use of header files can make a huge difference to the readability, size and performance of your code. The following rules will guide you through the various pitfalls of using header files. The #define Guard link ▶All header files should have #define guards to prevent multiple inclusion. The format of the symbol name should be ___H_. To guarantee uniqueness, they should be based on the full path in a project's source tree. For example, the file foo/src/bar/baz.h in project foo should have the following guard: #ifndef FOO_BAR_BAZ_H_ #define FOO_BAR_BAZ_H_ ... #endif // FOO_BAR_BAZ_H_ Header File Dependencies link ▶Don't use an #include when a forward declaration would suffice. When you include a header file you introduce a dependency that will cause your code to be recompiled whenever the header file changes. If your header file includes other header files, any change to those files will cause any code that includes your header to be recompiled. Therefore, we prefer to minimize includes, particularly includes of header files in other header files. You can significantly minimize the number of header files you need to include in your own header files by using forward declarations. For example, if your header file uses the File class in ways that do not require access to the declaration of the File class, your header file can just forward declare class File; instead of having to #include "file/base/file.h". How can we use a class Foo in a header file without access to its definition? We can declare data members of type Foo* or Foo&. We can declare (but not define) functions with arguments, and/or return values, of type Foo. (One exception is if an argument Foo or const Foo& has a non-explicit, one-argument constructor, in which case we need the full definition to support automatic type conversion.) We can declare static data members of type Foo. This is because static data members are defined outside the class definition. On the other hand, you must include the header file for Foo if your class subclasses Foo or has a data member of type Foo. Sometimes it makes sense to have pointer (or better, scoped_ptr) members instead of object members. However, this complicates code readability and imposes a performance penalty, so avoid doing this transformation if the only purpose is to minimize includes in header files. Of course, .cc files typically do require the definitions of the classes they use, and usually have to include several header files. Note: If you use a symbol Foo in your source file, you should bring in a definition for Foo yourself, either via an #include or via a forward declaration. Do not depend on the symbol being brought in transitively via headers not directly included. One exception is if Foo is used in myfile.cc, it's ok to #include (or forward-declare) Foo in myfile.h, instead of myfile.cc. Inline Functions link ▶Define functions inline only when they are small, say, 10 lines or less. Definition: You can declare functions in a way that allows the compiler to expand them inline rather than calling them through the usual function call mechanism. Pros: Inlining a function can generate more efficient object code, as long as the inlined function is small. Feel free to inline accessors and mutators, and other short, performance-critical functions. Cons: Overuse of inlining can actually make programs slower. Depending on a function's size, inlining it can cause the code size to increase or decrease. Inlining a very small accessor function will usually decrease code size while inlining a very large function can dramatically increase code size. On modern processors smaller code usually runs faster due to better use of the instruction cache. Decision: A decent rule of thumb is to not inline a function if it is more than 10 lines long. Beware of destructors, which are often longer than they appear because of implicit member- and base-destructor calls! Another useful rule of thumb: it's typically not cost effective to inline functions with loops or switch statements (unless, in the common case, the loop or switch statement is never executed). It is important to know that functions are not always inlined even if they are declared as such; for example, virtual and recursive functions are not normally inlined. Usually recursive functions should not be inline. The main reason for making a virtual function inline is to place its definition in the class, either for convenience or to document its behavior, e.g., for accessors and mutators. The -inl.h Files link ▶You may use file names with a -inl.h suffix to define complex inline functions when needed. The definition of an inline function needs to be in a header file, so that the compiler has the definition available for inlining at the call sites. However, implementation code properly belongs in .cc files, and we do not like to have much actual code in .h files unless there is a readability or performance advantage. If an inline function definition is short, with very little, if any, logic in it, you should put the code in your .h file. For example, accessors and mutators should certainly be inside a class definition. More complex inline functions may also be put in a .h file for the convenience of the implementer and callers, though if this makes the .h file too unwieldy you can instead put that code in a separate -inl.h file. This separates the implementation from the class definition, while still allowing the implementation to be included where necessary. Another use of -inl.h files is for definitions of function templates. This can be used to keep your template definitions easy to read. Do not forget that a -inl.h file requires a #define guard just like any other header file. Function Parameter Ordering link ▶When defining a function, parameter order is: inputs, then outputs. Parameters to C/C++ functions are either input to the function, output from the function, or both. Input parameters are usually values or const references, while output and input/output parameters will be non-const pointers. When ordering function parameters, put all input-only parameters before any output parameters. In particular, do not add new parameters to the end of the function just because they are new; place new input-only parameters before the output parameters. This is not a hard-and-fast rule. Parameters that are both input and output (often classes/structs) muddy the waters, and, as always, consistency with related functions may require you to bend the rule. Names and Order of Includes link ▶Use standard order for readability and to avoid hidden dependencies: C library, C++ library, other libraries' .h, your project's .h. All of a project's header files should be listed as descentants of the project's source directory without use of UNIX directory shortcuts . (the current directory) or .. (the parent directory). For example, google-awesome-project/src/base/logging.h should be included as #include "base/logging.h" In dir/foo.cc, whose main purpose is to implement or test the stuff in dir2/foo2.h, order your includes as follows: dir2/foo2.h (preferred location — see details below). C system files. C++ system files. Other libraries' .h files. Your project's .h files. The preferred ordering reduces hidden dependencies. We want every header file to be compilable on its own. The easiest way to achieve this is to make sure that every one of them is the first .h file #included in some .cc. dir/foo.cc and dir2/foo2.h are often in the same directory (e.g. base/basictypes_test.cc and base/basictypes.h), but can be in different directories too. Within each section it is nice to order the includes alphabetically. For example, the includes in google-awesome-project/src/foo/internal/fooserver.cc might look like this: #include "foo/public/fooserver.h" // Preferred location. #include #include #include #include #include "base/basictypes.h" #include "base/commandlineflags.h" #include "foo/public/bar.h" Scoping Namespaces link ▶Unnamed namespaces in .cc files are encouraged. With named namespaces, choose the name based on the project, and possibly its path. Do not use a using-directive. Definition: Namespaces subdivide the global scope into distinct, named scopes, and so are useful for preventing name collisions in the global scope. Pros: Namespaces provide a (hierarchical) axis of naming, in addition to the (also hierarchical) name axis provided by classes. For example, if two different projects have a class Foo in the global scope, these symbols may collide at compile time or at runtime. If each project places their code in a namespace, project1::Foo and project2::Foo are now distinct symbols that do not collide. Cons: Namespaces can be confusing, because they provide an additional (hierarchical) axis of naming, in addition to the (also hierarchical) name axis provided by classes. Use of unnamed spaces in header files can easily cause violations of the C++ One Definition Rule (ODR). Decision: Use namespaces according to the policy described below. Unnamed Namespaces Unnamed namespaces are allowed and even encouraged in .cc files, to avoid runtime naming conflicts: namespace { // This is in a .cc file. // The content of a namespace is not indented enum { kUnused, kEOF, kError }; // Commonly used tokens. bool AtEof() { return pos_ == kEOF; } // Uses our namespace's EOF. } // namespace However, file-scope declarations that are associated with a particular class may be declared in that class as types, static data members or static member functions rather than as members of an unnamed namespace. Terminate the unnamed namespace as shown, with a comment // namespace. Do not use unnamed namespaces in .h files. Named Namespaces Named namespaces should be used as follows: Namespaces wrap the entire source file after includes, gflags definitions/declarations, and forward declarations of classes from other namespaces: // In the .h file namespace mynamespace { // All declarations are within the namespace scope. // Notice the lack of indentation. class MyClass { public: ... void Foo(); }; } // namespace mynamespace // In the .cc file namespace mynamespace { // Definition of functions is within scope of the namespace. void MyClass::Foo() { ... } } // namespace mynamespace The typical .cc file might have more complex detail, including the need to reference classes in other namespaces. #include "a.h" DEFINE_bool(someflag, false, "dummy flag"); class C; // Forward declaration of class C in the global namespace. namespace a { class A; } // Forward declaration of a::A. namespace b { ...code for b... // Code goes against the left margin. } // namespace b Do not declare anything in namespace std, not even forward declarations of standard library classes. Declaring entities in namespace std is undefined behavior, i.e., not portable. To declare entities from the standard library, include the appropriate header file. You may not use a using-directive to make all names from a namespace available. // Forbidden -- This pollutes the namespace. using namespace foo; You may use a using-declaration anywhere in a .cc file, and in functions, methods or classes in .h files. // OK in .cc files. // Must be in a function, method or class in .h files. using ::foo::bar; Namespace aliases are allowed anywhere in a .cc file, anywhere inside the named namespace that wraps an entire .h file, and in functions and methods. // Shorten access to some commonly used names in .cc files. namespace fbz = ::foo::bar::baz; // Shorten access to some commonly used names (in a .h file). namespace librarian { // The following alias is available to all files including // this header (in namespace librarian): // alias names should therefore be chosen consistently // within a project. namespace pd_s = ::pipeline_diagnostics::sidetable; inline void my_inline_function() { // namespace alias local to a function (or method). namespace fbz = ::foo::bar::baz; ... } } // namespace librarian Note that an alias in a .h file is visible to everyone #including that file, so public headers (those available outside a project) and headers transitively #included by them, should avoid defining aliases, as part of the general goal of keeping public APIs as small as possible. Nested Classes link ▶Although you may use public nested classes when they are part of an interface, consider a namespace to keep declarations out of the global scope. Definition: A class can define another class within it; this is also called a member class. class Foo { private: // Bar is a member class, nested within Foo. class Bar { ... }; }; Pros: This is useful when the nested (or member) class is only used by the enclosing class; making it a member puts it in the enclosing class scope rather than polluting the outer scope with the class name. Nested classes can be forward declared within the enclosing class and then defined in the .cc file to avoid including the nested class definition in the enclosing class declaration, since the nested class definition is usually only relevant to the implementation. Cons: Nested classes can be forward-declared only within the definition of the enclosing class. Thus, any header file manipulating a Foo::Bar* pointer will have to include the full class declaration for Foo. Decision: Do not make nested classes public unless they are actually part of the interface, e.g., a class that holds a set of options for some method. Nonmember, Static Member, and Global Functions link ▶Prefer nonmember functions within a namespace or static member functions to global functions; use completely global functions rarely. Pros: Nonmember and static member functions can be useful in some situations. Putting nonmember functions in a namespace avoids polluting the global namespace. Cons: Nonmember and static member functions may make more sense as members of a new class, especially if they access external resources or have significant dependencies. Decision: Sometimes it is useful, or even necessary, to define a function not bound to a class instance. Such a function can be either a static member or a nonmember function. Nonmember functions should not depend on external variables, and should nearly always exist in a namespace. Rather than creating classes only to group static member functions which do not share static data, use namespaces instead. Functions defined in the same compilation unit as production classes may introduce unnecessary coupling and link-time dependencies when directly called from other compilation units; static member functions are particularly susceptible to this. Consider extracting a new class, or placing the functions in a namespace possibly in a separate library. If you must define a nonmember function and it is only needed in its .cc file, use an unnamed namespace or static linkage (eg static int Foo() {...}) to limit its scope. Local Variables link ▶Place a function's variables in the narrowest scope possible, and initialize variables in the declaration. C++ allows you to declare variables anywhere in a function. We encourage you to declare them in as local a scope as possible, and as close to the first use as possible. This makes it easier for the reader to find the declaration and see what type the variable is and what it was initialized to. In particular, initialization should be used instead of declaration and assignment, e.g. int i; i = f(); // Bad -- initialization separate from declaration. int j = g(); // Good -- declaration has initialization. Note that gcc implements for (int i = 0; i < 10; ++i) correctly (the scope of i is only the scope of the for loop), so you can then reuse i in another for loop in the same scope. It also correctly scopes declarations in if and while statements, e.g. while (const char* p = strchr(str, '/')) str = p + 1; There is one caveat: if the variable is an object, its constructor is invoked every time it enters scope and is created, and its destructor is invoked every time it goes out of scope. // Inefficient implementation: for (int i = 0; i < 1000000; ++i) { Foo f; // My ctor and dtor get called 1000000 times each. f.DoSomething(i); } It may be more efficient to declare such a variable used in a loop outside that loop: Foo f; // My ctor and dtor get called once each. for (int i = 0; i < 1000000; ++i) { f.DoSomething(i); } Static and Global Variables link ▶Static or global variables of class type are forbidden: they cause hard-to-find bugs due to indeterminate order of construction and destruction. Objects with static storage duration, including global variables, static variables, static class member variables, and function static variables, must be Plain Old Data (POD): only ints, chars, floats, or pointers, or arrays/structs of POD. The order in which class constructors and initializers for static variables are called is only partially specified in C++ and can even change from build to build, which can cause bugs that are difficult to find. Therefore in addition to banning globals of class type, we do not allow static POD variables to be initialized with the result of a function, unless that function (such as getenv(), or getpid()) does not itself depend on any other globals. Likewise, the order in which destructors are called is defined to be the reverse of the order in which the constructors were called. Since constructor order is indeterminate, so is destructor order. For example, at program-end time a static variable might have been destroyed, but code still running -- perhaps in another thread -- tries to access it and fails. Or the destructor for a static 'string' variable might be run prior to the destructor for another variable that contains a reference to that string. As a result we only allow static variables to contain POD data. This rule completely disallows vector (use C arrays instead), or string (use const char []). If you need a static or global variable of a class type, consider initializing a pointer (which will never be freed), from either your main() function or from pthread_once(). Note that this must be a raw pointer, not a "smart" pointer, since the smart pointer's destructor will have the order-of-destructor issue that we are trying to avoid. Classes Classes are the fundamental unit of code in C++. Naturally, we use them extensively. This section lists the main dos and don'ts you should follow when writing a class. Doing Work in Constructors link ▶In general, constructors should merely set member variables to their initial values. Any complex initialization should go in an explicit Init() method. Definition: It is possible to perform initialization in the body of the constructor. Pros: Convenience in typing. No need to worry about whether the class has been initialized or not. Cons: The problems with doing work in constructors are: There is no easy way for constructors to signal errors, short of using exceptions (which are forbidden). If the work fails, we now have an object whose initialization code failed, so it may be an indeterminate state. If the work calls virtual functions, these calls will not get dispatched to the subclass implementations. Future modification to your class can quietly introduce this problem even if your class is not currently subclassed, causing much confusion. If someone creates a global variable of this type (which is against the rules, but still), the constructor code will be called before main(), possibly breaking some implicit assumptions in the constructor code. For instance, gflags will not yet have been initialized. Decision: If your object requires non-trivial initialization, consider having an explicit Init() method. In particular, constructors should not call virtual functions, attempt to raise errors, access potentially uninitialized global variables, etc. Default Constructors link ▶You must define a default constructor if your class defines member variables and has no other constructors. Otherwise the compiler will do it for you, badly. Definition: The default constructor is called when we new a class object with no arguments. It is always called when calling new[] (for arrays). Pros: Initializing structures by default, to hold "impossible" values, makes debugging much easier. Cons: Extra work for you, the code writer. Decision: If your class defines member variables and has no other constructors you must define a default constructor (one that takes no arguments). It should preferably initialize the object in such a way that its internal state is consistent and valid. The reason for this is that if you have no other constructors and do not define a default constructor, the compiler will generate one for you. This compiler generated constructor may not initialize your object sensibly. If your class inherits from an existing class but you add no new member variables, you are not required to have a default constructor. Explicit Constructors link ▶Use the C++ keyword explicit for constructors with one argument. Definition: Normally, if a constructor takes one argument, it can be used as a conversion. For instance, if you define Foo::Foo(string name) and then pass a string to a function that expects a Foo, the constructor will be called to convert the string into a Foo and will pass the Foo to your function for you. This can be convenient but is also a source of trouble when things get converted and new objects created without you meaning them to. Declaring a constructor explicit prevents it from being invoked implicitly as a conversion. Pros: Avoids undesirable conversions. Cons: None. Decision: We require all single argument constructors to be explicit. Always put explicit in front of one-argument constructors in the class definition: explicit Foo(string name); The exception is copy constructors, which, in the rare cases when we allow them, should probably not be explicit. Classes that are intended to be transparent wrappers around other classes are also exceptions. Such exceptions should be clearly marked with comments. Copy Constructors link ▶Provide a copy constructor and assignment operator only when necessary. Otherwise, disable them with DISALLOW_COPY_AND_ASSIGN. Definition: The copy constructor and assignment operator are used to create copies of objects. The copy constructor is implicitly invoked by the compiler in some situations, e.g. passing objects by value. Pros: Copy constructors make it easy to copy objects. STL containers require that all contents be copyable and assignable. Copy constructors can be more efficient than CopyFrom()-style workarounds because they combine construction with copying, the compiler can elide them in some contexts, and they make it easier to avoid heap allocation. Cons: Implicit copying of objects in C++ is a rich source of bugs and of performance problems. It also reduces readability, as it becomes hard to track which objects are being passed around by value as opposed to by reference, and therefore where changes to an object are reflected. Decision: Few classes need to be copyable. Most should have neither a copy constructor nor an assignment operator. In many situations, a pointer or reference will work just as well as a copied value, with better performance. For example, you can pass function parameters by reference or pointer instead of by value, and you can store pointers rather than objects in an STL container. If your class needs to be copyable, prefer providing a copy method, such as CopyFrom() or Clone(), rather than a copy constructor, because such methods cannot be invoked implicitly. If a copy method is insufficient in your situation (e.g. for performance reasons, or because your class needs to be stored by value in an STL container), provide both a copy constructor and assignment operator. If your class does not need a copy constructor or assignment operator, you must explicitly disable them. To do so, add dummy declarations for the copy constructor and assignment operator in the private: section of your class, but do not provide any corresponding definition (so that any attempt to use them results in a link error). For convenience, a DISALLOW_COPY_AND_ASSIGN macro can be used: // A macro to disallow the copy constructor and operator= functions // This should be used in the private: declarations for a class #define DISALLOW_COPY_AND_ASSIGN(TypeName) \ TypeName(const TypeName&); \ void operator=(const TypeName&) Then, in class Foo: class Foo { public: Foo(int f); ~Foo(); private: DISALLOW_COPY_AND_ASSIGN(Foo); }; Structs vs. Classes link ▶Use a struct only for passive objects that carry data; everything else is a class. The struct and class keywords behave almost identically in C++. We add our own semantic meanings to each keyword, so you should use the appropriate keyword for the data-type you're defining. structs should be used for passive objects that carry data, and may have associated constants, but lack any functionality other than access/setting the data members. The accessing/setting of fields is done by directly accessing the fields rather than through method invocations. Methods should not provide behavior but should only be used to set up the data members, e.g., constructor, destructor, Initialize(), Reset(), Validate(). If more functionality is required, a class is more appropriate. If in doubt, make it a class. For consistency with STL, you can use struct instead of class for functors and traits. Note that member variables in structs and classes have different naming rules. Inheritance link ▶Composition is often more appropriate than inheritance. When using inheritance, make it public. Definition: When a sub-class inherits from a base class, it includes the definitions of all the data and operations that the parent base class defines. In practice, inheritance is used in two major ways in C++: implementation inheritance, in which actual code is inherited by the child, and interface inheritance, in which only method names are inherited. Pros: Implementation inheritance reduces code size by re-using the base class code as it specializes an existing type. Because inheritance is a compile-time declaration, you and the compiler can understand the operation and detect errors. Interface inheritance can be used to programmatically enforce that a class expose a particular API. Again, the compiler can detect errors, in this case, when a class does not define a necessary method of the API. Cons: For implementation inheritance, because the code implementing a sub-class is spread between the base and the sub-class, it can be more difficult to understand an implementation. The sub-class cannot override functions that are not virtual, so the sub-class cannot change implementation. The base class may also define some data members, so that specifies physical layout of the base class. Decision: All inheritance should be public. If you want to do private inheritance, you should be including an instance of the base class as a member instead. Do not overuse implementation inheritance. Composition is often more appropriate. Try to restrict use of inheritance to the "is-a" case: Bar subclasses Foo if it can reasonably be said that Bar "is a kind of" Foo. Make your destructor virtual if necessary. If your class has virtual methods, its destructor should be virtual. Limit the use of protected to those member functions that might need to be accessed from subclasses. Note that data members should be private. When redefining an inherited virtual function, explicitly declare it virtual in the declaration of the derived class. Rationale: If virtual is omitted, the reader has to check all ancestors of the class in question to determine if the function is virtual or not. Multiple Inheritance link ▶Only very rarely is multiple implementation inheritance actually useful. We allow multiple inheritance only when at most one of the base classes has an implementation; all other base classes must be pure interface classes tagged with the Interface suffix. Definition: Multiple inheritance allows a sub-class to have more than one base class. We distinguish between base classes that are pure interfaces and those that have an implementation. Pros: Multiple implementation inheritance may let you re-use even more code than single inheritance (see Inheritance). Cons: Only very rarely is multiple implementation inheritance actually useful. When multiple implementation inheritance seems like the solution, you can usually find a different, more explicit, and cleaner solution. Decision: Multiple inheritance is allowed only when all superclasses, with the possible exception of the first one, are pure interfaces. In order to ensure that they remain pure interfaces, they must end with the Interface suffix. Note: There is an exception to this rule on Windows. Interfaces link ▶Classes that satisfy certain conditions are allowed, but not required, to end with an Interface suffix. Definition: A class is a pure interface if it meets the following requirements: It has only public pure virtual ("= 0") methods and static methods (but see below for destructor). It may not have non-static data members. It need not have any constructors defined. If a constructor is provided, it must take no arguments and it must be protected. If it is a subclass, it may only be derived from classes that satisfy these conditions and are tagged with the Interface suffix. An interface class can never be directly instantiated because of the pure virtual method(s) it declares. To make sure all implementations of the interface can be destroyed correctly, they must also declare a virtual destructor (in an exception to the first rule, this should not be pure). See Stroustrup, The C++ Programming Language, 3rd edition, section 12.4 for details. Pros: Tagging a class with the Interface suffix lets others know that they must not add implemented methods or non static data members. This is particularly important in the case of multiple inheritance. Additionally, the interface concept is already well-understood by Java programmers. Cons: The Interface suffix lengthens the class name, which can make it harder to read and understand. Also, the interface property may be considered an implementation detail that shouldn't be exposed to clients. Decision: A class may end with Interface only if it meets the above requirements. We do not require the converse, however: classes that meet the above requirements are not required to end with Interface. Operator Overloading link ▶Do not overload operators except in rare, special circumstances. Definition: A class can define that operators such as + and / operate on the class as if it were a built-in type. Pros: Can make code appear more intuitive because a class will behave in the same way as built-in types (such as int). Overloaded operators are more playful names for functions that are less-colorfully named, such as Equals() or Add(). For some template functions to work correctly, you may need to define operators. Cons: While operator overloading can make code more intuitive, it has several drawbacks: It can fool our intuition into thinking that expensive operations are cheap, built-in operations. It is much harder to find the call sites for overloaded operators. Searching for Equals() is much easier than searching for relevant invocations of ==. Some operators work on pointers too, making it easy to introduce bugs. Foo + 4 may do one thing, while &Foo + 4 does something totally different. The compiler does not complain for either of these, making this very hard to debug. Overloading also has surprising ramifications. For instance, if a class overloads unary operator&, it cannot safely be forward-declared. Decision: In general, do not overload operators. The assignment operator (operator=), in particular, is insidious and should be avoided. You can define functions like Equals() and CopyFrom() if you need them. Likewise, avoid the dangerous unary operator& at all costs, if there's any possibility the class might be forward-declared. However, there may be rare cases where you need to overload an operator to interoperate with templates or "standard" C++ classes (such as operator<<(ostream&, const T&) for logging). These are acceptable if fully justified, but you should try to avoid these whenever possible. In particular, do not overload operator== or operator< just so that your class can be used as a key in an STL container; instead, you should create equality and comparison functor types when declaring the container. Some of the STL algorithms do require you to overload operator==, and you may do so in these cases, provided you document why. See also Copy Constructors and Function Overloading. Access Control link ▶Make data members private, and provide access to them through accessor functions as needed (for technical reasons, we allow data members of a test fixture class to be protected when using Google Test). Typically a variable would be called foo_ and the accessor function foo(). You may also want a mutator function set_foo(). Exception: static const data members (typically called kFoo) need not be private. The definitions of accessors are usually inlined in the header file. See also Inheritance and Function Names. Declaration Order link ▶Use the specified order of declarations within a class: public: before private:, methods before data members (variables), etc. Your class definition should start with its public: section, followed by its protected: section and then its private: section. If any of these sections are empty, omit them. Within each section, the declarations generally should be in the following order: Typedefs and Enums Constants (static const data members) Constructors Destructor Methods, including static methods Data Members (except static const data members) Friend declarations should always be in the private section, and the DISALLOW_COPY_AND_ASSIGN macro invocation should be at the end of the private: section. It should be the last thing in the class. See Copy Constructors. Method definitions in the corresponding .cc file should be the same as the declaration order, as much as possible. Do not put large method definitions inline in the class definition. Usually, only trivial or performance-critical, and very short, methods may be defined inline. See Inline Functions for more details. Write Short Functions link ▶Prefer small and focused functions. We recognize that long functions are sometimes appropriate, so no hard limit is placed on functions length. If a function exceeds about 40 lines, think about whether it can be broken up without harming the structure of the program. Even if your long function works perfectly now, someone modifying it in a few months may add new behavior. This could result in bugs that are hard to find. Keeping your functions short and simple makes it easier for other people to read and modify your code. You could find long and complicated functions when working with some code. Do not be intimidated by modifying existing code: if working with such a function proves to be difficult, you find that errors are hard to debug, or you want to use a piece of it in several different contexts, consider breaking up the function into smaller and more manageable pieces. Google-Specific Magic There are various tricks and utilities that we use to make C++ code more robust, and various ways we use C++ that may differ from what you see elsewhere. Smart Pointers link ▶If you actually need pointer semantics, scoped_ptr is great. You should only use std::tr1::shared_ptr under very specific conditions, such as when objects need to be held by STL containers. You should never use auto_ptr. "Smart" pointers are objects that act like pointers but have added semantics. When a scoped_ptr is destroyed, for instance, it deletes the object it's pointing to. shared_ptr is the same way, but implements reference-counting so only the last pointer to an object deletes it. Generally speaking, we prefer that we design code with clear object ownership. The clearest object ownership is obtained by using an object directly as a field or local variable, without using pointers at all. On the other extreme, by their very definition, reference counted pointers are owned by nobody. The problem with this design is that it is easy to create circular references or other strange conditions that cause an object to never be deleted. It is also slow to perform atomic operations every time a value is copied or assigned. Although they are not recommended, reference counted pointers are sometimes the simplest and most elegant way to solve a problem. cpplint link ▶Use cpplint.py to detect style errors. cpplint.py is a tool that reads a source file and identifies many style errors. It is not perfect, and has both false positives and false negatives, but it is still a valuable tool. False positives can be ignored by putting // NOLINT at the end of the line. Some projects have instructions on how to run cpplint.py from their project tools. If the project you are contributing to does not, you can download cpplint.py separately. Other C++ Features Reference Arguments link ▶All parameters passed by reference must be labeled const. Definition: In C, if a function needs to modify a variable, the parameter must use a pointer, eg int foo(int *pval). In C++, the function can alternatively declare a reference parameter: int foo(int &val). Pros: Defining a parameter as reference avoids ugly code like (*pval)++. Necessary for some applications like copy constructors. Makes it clear, unlike with pointers, that NULL is not a possible value. Cons: References can be confusing, as they have value syntax but pointer semantics. Decision: Within function parameter lists all references must be const: void Foo(const string &in, string *out); In fact it is a very strong convention in Google code that input arguments are values or const references while output arguments are pointers. Input parameters may be const pointers, but we never allow non-const reference parameters. One case when you might want an input parameter to be a const pointer is if you want to emphasize that the argument is not copied, so it must exist for the lifetime of the object; it is usually best to document this in comments as well. STL adapters such as bind2nd and mem_fun do not permit reference parameters, so you must declare functions with pointer parameters in these cases, too. Function Overloading link ▶Use overloaded functions (including constructors) only if a reader looking at a call site can get a good idea of what is happening without having to first figure out exactly which overload is being called. Definition: You may write a function that takes a const string& and overload it with another that takes const char*. class MyClass { public: void Analyze(const string &text); void Analyze(const char *text, size_t textlen); }; Pros: Overloading can make code more intuitive by allowing an identically-named function to take different arguments. It may be necessary for templatized code, and it can be convenient for Visitors. Cons: If a function is overloaded by the argument types alone, a reader may have to understand C++'s complex matching rules in order to tell what's going on. Also many people are confused by the semantics of inheritance if a derived class overrides only some of the variants of a function. Decision: If you want to overload a function, consider qualifying the name with some information about the arguments, e.g., AppendString(), AppendInt() rather than just Append(). Default Arguments link ▶We do not allow default function parameters, except in a few uncommon situations explained below. Pros: Often you have a function that uses lots of default values, but occasionally you want to override the defaults. Default parameters allow an easy way to do this without having to define many functions for the rare exceptions. Cons: People often figure out how to use an API by looking at existing code that uses it. Default parameters are more difficult to maintain because copy-and-paste from previous code may not reveal all the parameters. Copy-and-pasting of code segments can cause major problems when the default arguments are not appropriate for the new code. Decision: Except as described below, we require all arguments to be explicitly specified, to force programmers to consider the API and the values they are passing for each argument rather than silently accepting defaults they may not be aware of. One specific exception is when default arguments are used to simulate variable-length argument lists. // Support up to 4 params by using a default empty AlphaNum. string StrCat(const AlphaNum &a, const AlphaNum &b = gEmptyAlphaNum, const AlphaNum &c = gEmptyAlphaNum, const AlphaNum &d = gEmptyAlphaNum); Variable-Length Arrays and alloca() link ▶We do not allow variable-length arrays or alloca(). Pros: Variable-length arrays have natural-looking syntax. Both variable-length arrays and alloca() are very efficient. Cons: Variable-length arrays and alloca are not part of Standard C++. More importantly, they allocate a data-dependent amount of stack space that can trigger difficult-to-find memory overwriting bugs: "It ran fine on my machine, but dies mysteriously in production". Decision: Use a safe allocator instead, such as scoped_ptr/scoped_array. Friends link ▶We allow use of friend classes and functions, within reason. Friends should usually be defined in the same file so that the reader does not have to look in another file to find uses of the private members of a class. A common use of friend is to have a FooBuilder class be a friend of Foo so that it can construct the inner state of Foo correctly, without exposing this state to the world. In some cases it may be useful to make a unittest class a friend of the class it tests. Friends extend, but do not break, the encapsulation boundary of a class. In some cases this is better than making a member public when you want to give only one other class access to it. However, most classes should interact with other classes solely through their public members. Exceptions link ▶We do not use C++ exceptions. Pros: Exceptions allow higher levels of an application to decide how to handle "can't happen" failures in deeply nested functions, without the obscuring and error-prone bookkeeping of error codes. Exceptions are used by most other modern languages. Using them in C++ would make it more consistent with Python, Java, and the C++ that others are familiar with. Some third-party C++ libraries use exceptions, and turning them off internally makes it harder to integrate with those libraries. Exceptions are the only way for a constructor to fail. We can simulate this with a factory function or an Init() method, but these require heap allocation or a new "invalid" state, respectively. Exceptions are really handy in testing frameworks. Cons: When you add a throw statement to an existing function, you must examine all of its transitive callers. Either they must make at least the basic exception safety guarantee, or they must never catch the exception and be happy with the program terminating as a result. For instance, if f() calls g() calls h(), and h throws an exception that f catches, g has to be careful or it may not clean up properly. More generally, exceptions make the control flow of programs difficult to evaluate by looking at code: functions may return in places you don't expect. This causes maintainability and debugging difficulties. You can minimize this cost via some rules on how and where exceptions can be used, but at the cost of more that a developer needs to know and understand. Exception safety requires both RAII and different coding practices. Lots of supporting machinery is needed to make writing correct exception-safe code easy. Further, to avoid requiring readers to understand the entire call graph, exception-safe code must isolate logic that writes to persistent state into a "commit" phase. This will have both benefits and costs (perhaps where you're forced to obfuscate code to isolate the commit). Allowing exceptions would force us to always pay those costs even when they're not worth it. Turning on exceptions adds data to each binary produced, increasing compile time (probably slightly) and possibly increasing address space pressure. The availability of exceptions may encourage developers to throw them when they are not appropriate or recover from them when it's not safe to do so. For example, invalid user input should not cause exceptions to be thrown. We would need to make the style guide even longer to document these restrictions! Decision: On their face, the benefits of using exceptions outweigh the costs, especially in new projects. However, for existing code, the introduction of exceptions has implications on all dependent code. If exceptions can be propagated beyond a new project, it also becomes problematic to integrate the new project into existing exception-free code. Because most existing C++ code at Google is not prepared to deal with exceptions, it is comparatively difficult to adopt new code that generates exceptions. Given that Google's existing code is not exception-tolerant, the costs of using exceptions are somewhat greater than the costs in a new project. The conversion process would be slow and error-prone. We don't believe that the available alternatives to exceptions, such as error codes and assertions, introduce a significant burden. Our advice against using exceptions is not predicated on philosophical or moral grounds, but practical ones. Because we'd like to use our open-source projects at Google and it's difficult to do so if those projects use exceptions, we need to advise against exceptions in Google open-source projects as well. Things would probably be different if we had to do it all over again from scratch. There is an exception to this rule (no pun intended) for Windows code. Run-Time Type Information (RTTI) link ▶We do not use Run Time Type Information (RTTI). Definition: RTTI allows a programmer to query the C++ class of an object at run time. Pros: It is useful in some unittests. For example, it is useful in tests of factory classes where the test has to verify that a newly created object has the expected dynamic type. In rare circumstances, it is useful even outside of tests. Cons: A query of type during run-time typically means a design problem. If you need to know the type of an object at runtime, that is often an indication that you should reconsider the design of your class. Decision: Do not use RTTI, except in unittests. If you find yourself in need of writing code that behaves differently based on the class of an object, consider one of the alternatives to querying the type. Virtual methods are the preferred way of executing different code paths depending on a specific subclass type. This puts the work within the object itself. If the work belongs outside the object and instead in some processing code, consider a double-dispatch solution, such as the Visitor design pattern. This allows a facility outside the object itself to determine the type of class using the built-in type system. If you think you truly cannot use those ideas, you may use RTTI. But think twice about it. :-) Then think twice again. Do not hand-implement an RTTI-like workaround. The arguments against RTTI apply just as much to workarounds like class hierarchies with type tags. Casting link ▶Use C++ casts like static_cast(). Do not use other cast formats like int y = (int)x; or int y = int(x);. Definition: C++ introduced a different cast system from C that distinguishes the types of cast operations. Pros: The problem with C casts is the ambiguity of the operation; sometimes you are doing a conversion (e.g., (int)3.5) and sometimes you are doing a cast (e.g., (int)"hello"); C++ casts avoid this. Additionally C++ casts are more visible when searching for them. Cons: The syntax is nasty. Decision: Do not use C-style casts. Instead, use these C++-style casts. Use static_cast as the equivalent of a C-style cast that does value conversion, or when you need to explicitly up-cast a pointer from a class to its superclass. Use const_cast to remove the const qualifier (see const). Use reinterpret_cast to do unsafe conversions of pointer types to and from integer and other pointer types. Use this only if you know what you are doing and you understand the aliasing issues. Do not use dynamic_cast except in test code. If you need to know type information at runtime in this way outside of a unittest, you probably have a design flaw. Streams link ▶Use streams only for logging. Definition: Streams are a replacement for printf() and scanf(). Pros: With streams, you do not need to know the type of the object you are printing. You do not have problems with format strings not matching the argument list. (Though with gcc, you do not have that problem with printf either.) Streams have automatic constructors and destructors that open and close the relevant files. Cons: Streams make it difficult to do functionality like pread(). Some formatting (particularly the common format string idiom %.*s) is difficult if not impossible to do efficiently using streams without using printf-like hacks. Streams do not support operator reordering (the %1s directive), which is helpful for internationalization. Decision: Do not use streams, except where required by a logging interface. Use printf-like routines instead. There are various pros and cons to using streams, but in this case, as in many other cases, consistency trumps the debate. Do not use streams in your code. Extended Discussion There has been debate on this issue, so this explains the reasoning in greater depth. Recall the Only One Way guiding principle: we want to make sure that whenever we do a certain type of I/O, the code looks the same in all those places. Because of this, we do not want to allow users to decide between using streams or using printf plus Read/Write/etc. Instead, we should settle on one or the other. We made an exception for logging because it is a pretty specialized application, and for historical reasons. Proponents of streams have argued that streams are the obvious choice of the two, but the issue is not actually so clear. For every advantage of streams they point out, there is an equivalent disadvantage. The biggest advantage is that you do not need to know the type of the object to be printing. This is a fair point. But, there is a downside: you can easily use the wrong type, and the compiler will not warn you. It is easy to make this kind of mistake without knowing when using streams. cout << this; // Prints the address cout << *this; // Prints the contents The compiler does not generate an error because << has been overloaded. We discourage overloading for just this reason. Some say printf formatting is ugly and hard to read, but streams are often no better. Consider the following two fragments, both with the same typo. Which is easier to discover? cerr << "Error connecting to '" hostname.first << ":" hostname.second << ": " hostname.first, foo->bar()->hostname.second, strerror(errno)); And so on and so forth for any issue you might bring up. (You could argue, "Things would be better with the right wrappers," but if it is true for one scheme, is it not also true for the other? Also, remember the goal is to make the language smaller, not add yet more machinery that someone has to learn.) Either path would yield different advantages and disadvantages, and there is not a clearly superior solution. The simplicity doctrine mandates we settle on one of them though, and the majority decision was on printf + read/write. Preincrement and Predecrement link ▶Use prefix form (++i) of the increment and decrement operators with iterators and other template objects. Definition: When a variable is incremented (++i or i++) or decremented (--i or i--) and the value of the expression is not used, one must decide whether to preincrement (decrement) or postincrement (decrement). Pros: When the return value is ignored, the "pre" form (++i) is never less efficient than the "post" form (i++), and is often more efficient. This is because post-increment (or decrement) requires a copy of i to be made, which is the value of the expression. If i is an iterator or other non-scalar type, copying i could be expensive. Since the two types of increment behave the same when the value is ignored, why not just always pre-increment? Cons: The tradition developed, in C, of using post-increment when the expression value is not used, especially in for loops. Some find post-increment easier to read, since the "subject" (i) precedes the "verb" (++), just like in English. Decision: For simple scalar (non-object) values there is no reason to prefer one form and we allow either. For iterators and other template types, use pre-increment. Use of const link ▶We strongly recommend that you use const whenever it makes sense to do so. Definition: Declared variables and parameters can be preceded by the keyword const to indicate the variables are not changed (e.g., const int foo). Class functions can have the const qualifier to indicate the function does not change the state of the class member variables (e.g., class Foo { int Bar(char c) const; };). Pros: Easier for people to understand how variables are being used. Allows the compiler to do better type checking, and, conceivably, generate better code. Helps people convince themselves of program correctness because they know the functions they call are limited in how they can modify your variables. Helps people know what functions are safe to use without locks in multi-threaded programs. Cons: const is viral: if you pass a const variable to a function, that function must have const in its prototype (or the variable will need a const_cast). This can be a particular problem when calling library functions. Decision: const variables, data members, methods and arguments add a level of compile-time type checking; it is better to detect errors as soon as possible. Therefore we strongly recommend that you use const whenever it makes sense to do so: If a function does not modify an argument passed by reference or by pointer, that argument should be const. Declare methods to be const whenever possible. Accessors should almost always be const. Other methods should be const if they do not modify any data members, do not call any non-const methods, and do not return a non-const pointer or non-const reference to a data member. Consider making data members const whenever they do not need to be modified after construction. However, do not go crazy with const. Something like const int * const * const x; is likely overkill, even if it accurately describes how const x is. Focus on what's really useful to know: in this case, const int** x is probably sufficient. The mutable keyword is allowed but is unsafe when used with threads, so thread safety should be carefully considered first. Where to put the const Some people favor the form int const *foo to const int* foo. They argue that this is more readable because it's more consistent: it keeps the rule that const always follows the object it's describing. However, this consistency argument doesn't apply in this case, because the "don't go crazy" dictum eliminates most of the uses you'd have to be consistent with. Putting the const first is arguably more readable, since it follows English in putting the "adjective" (const) before the "noun" (int). That said, while we encourage putting const first, we do not require it. But be consistent with the code around you! Integer Types link ▶Of the built-in C++ integer types, the only one used is int. If a program needs a variable of a different size, use a precise-width integer type from , such as int16_t. Definition: C++ does not specify the sizes of its integer types. Typically people assume that short is 16 bits, int is 32 bits, long is 32 bits and long long is 64 bits. Pros: Uniformity of declaration. Cons: The sizes of integral types in C++ can vary based on compiler and architecture. Decision: defines types like int16_t, uint32_t, int64_t, etc. You should always use those in preference to short, unsigned long long and the like, when you need a guarantee on the size of an integer. Of the C integer types, only int should be used. When appropriate, you are welcome to use standard types like size_t and ptrdiff_t. We use int very often, for integers we know are not going to be too big, e.g., loop counters. Use plain old int for such things. You should assume that an int is at least 32 bits, but don't assume that it has more than 32 bits. If you need a 64-bit integer type, use int64_t or uint64_t. For integers we know can be "big", use int64_t. You should not use the unsigned integer types such as uint32_t, unless the quantity you are representing is really a bit pattern rather than a number, or unless you need defined twos-complement overflow. In particular, do not use unsigned types to say a number will never be negative. Instead, use assertions for this. On Unsigned Integers Some people, including some textbook authors, recommend using unsigned types to represent numbers that are never negative. This is intended as a form of self-documentation. However, in C, the advantages of such documentation are outweighed by the real bugs it can introduce. Consider: for (unsigned int i = foo.Length()-1; i >= 0; --i) ... This code will never terminate! Sometimes gcc will notice this bug and warn you, but often it will not. Equally bad bugs can occur when comparing signed and unsigned variables. Basically, C's type-promotion scheme causes unsigned types to behave differently than one might expect. So, document that a variable is non-negative using assertions. Don't use an unsigned type. 64-bit Portability link ▶Code should be 64-bit and 32-bit friendly. Bear in mind problems of printing, comparisons, and structure alignment. printf() specifiers for some types are not cleanly portable between 32-bit and 64-bit systems. C99 defines some portable format specifiers. Unfortunately, MSVC 7.1 does not understand some of these specifiers and the standard is missing a few, so we have to define our own ugly versions in some cases (in the style of the standard include file inttypes.h): // printf macros for size_t, in the style of inttypes.h #ifdef _LP64 #define __PRIS_PREFIX "z" #else #define __PRIS_PREFIX #endif // Use these macros after a % in a printf format string // to get correct 32/64 bit behavior, like this: // size_t size = records.size(); // printf("%"PRIuS"\n", size); #define PRIdS __PRIS_PREFIX "d" #define PRIxS __PRIS_PREFIX "x" #define PRIuS __PRIS_PREFIX "u" #define PRIXS __PRIS_PREFIX "X" #define PRIoS __PRIS_PREFIX "o" Type DO NOT use DO use Notes void * (or any pointer) %lx %p int64_t %qd, %lld %"PRId64" uint64_t %qu, %llu, %llx %"PRIu64", %"PRIx64" size_t %u %"PRIuS", %"PRIxS" C99 specifies %zu ptrdiff_t %d %"PRIdS" C99 specifies %zd Note that the PRI* macros expand to independent strings which are concatenated by the compiler. Hence if you are using a non-constant format string, you need to insert the value of the macro into the format, rather than the name. It is still possible, as usual, to include length specifiers, etc., after the % when using the PRI* macros. So, e.g. printf("x = %30"PRIuS"\n", x) would expand on 32-bit Linux to printf("x = %30" "u" "\n", x), which the compiler will treat as printf("x = %30u\n", x). Remember that sizeof(void *) != sizeof(int). Use intptr_t if you want a pointer-sized integer. You may need to be careful with structure alignments, particularly for structures being stored on disk. Any class/structure with a int64_t/uint64_t member will by default end up being 8-byte aligned on a 64-bit system. If you have such structures being shared on disk between 32-bit and 64-bit code, you will need to ensure that they are packed the same on both architectures. Most compilers offer a way to alter structure alignment. For gcc, you can use __attribute__((packed)). MSVC offers #pragma pack() and __declspec(align()). Use the LL or ULL suffixes a
Changes in 2.4.6 (February 22, 2011): Brief summary : - Support more host OS to run on: - Include win64 native binary in the release. - Fixed failures on big endian hosts. - BIOS: Support for up to 2M ROM BIOS images. - GUI: select mouse capture toggle method in .bochsrc. - Ported most of Qemu's 'virtual VFAT' block driver (except runtime write support, but plus FAT32 suppport) - Added write protect option for floppy drives. - Bugfixes / improved internal debugger + instrumentation. Detailed change log : - CPU and internal debugger - Implemented Process Context ID (PCID) feature - Implemented FS/GS BASE access instructions support (according to document from http://software.intel.com/en-us/avx/) - Rewritten from scratch SMC detection algorithm - Implemented fine-grained SMC detection (on 128 byte granularity) - Bugfixes for CPU emulation correctness and stability - Fixed failures on Big Endian hosts ! - Print detailed page walk information and attributes in internal debugger 'page' command - Updated/Fixed instrumentation callbacks - Configure and compile - Bochs now can be compiled as native Windows x86-64 application (tested with Mingw gcc 4.5.1 and Microsoft Visual Studio Express 2010) - Added ability to configure CPUID stepping through .bochsrc. The default stepping value is 3. - Added ability to disable MONITOR/MWAIT support through .bochsrc CPUID option. The option is available only if compiled with --enable-monitor-mwait configure option. - Determine and select max physical address size automatically at configure time: - 32-bit physical address for 386/486 guests - 36-bit physical address for PSE-36 enabled Pentium guest - 40-bit physical address for PAE enabled P6 or later guests - Update config.guess/config.sub scripts to May 2010 revisions. - Update Visual Studio 2008 project files in build/win32/vs2008ex-workspace.zip - Added Bochs compilation timestamp after Bochs version string. - GUI and display libraries (Volker) - Added new .bochsrc option to select mouse capture toggle method. In addition to the default Bochs method using the CTRL key and the middle mouse button there are now the choices: - CTRL+F10 (like DOSBox) - CTRL+ALT (like QEMU) - F12 (replaces win32 'legacyF12' option) - display library 'x' now uses the desktop size for the maximum guest resolution - ROM BIOS - Support for up to 2M ROM BIOS images - I/O Devices - 3 new 'pseudo device' plugins created by plugin separation (see below) - Fixes for emulated DHCP in eth_vnet (patch from @SF tracker) - Added support for VGA graphics mode with 400 lines (partial fix for SF bug #2948724) - NE2K: Fixed "send buffer" command issue on big endian hosts - USB - converted common USB code plus devices to the new 'usb_common' plugin Now the USB device classes no longer exist twice if both HC plugins are loaded. - added 'pseudo device' in common USB code for the device creation. This makes the HCs independent from the device specific code. - USB MSD: added support for disk image modes (like ATA disks) - USB printer: output file creation failure now causes a disconnect - re-implemented "options" parameter for additional options of connected devices (currently only used to set the speed reported by device and to specify an alternative redolog file of USB MSD disk image modes) - hard drive - new disk image mode 'vvfat' - ported the read-only part of Qemu's 'virtual VFAT' block driver - additions: configurable disk geometry, FAT32 support, read MBR and/or boot sector from file, volatile write support using hdimage redolog_t class, optional commit support on Bochs exit, save/restore file attributes, 1.44 MB floppy support, set file modification date/time - converted the complete hdimage stuff to the new 'hdimage' plugin - new hdimage method get_capabilities() that can return special flags - vmware3, vmware4 and vvfat classes now return HDIMAGE_HAS_GEOMETRY flag - other disk image modes by default return HDIMAGE_AUTO_GEOMETRY if cylinder value is set to 0 - multiple sector read/write support for some image modes - new log prefix "IMG" for hdimage messages - floppy - added write protect option for floppy drives (based on @SF patch by Ben Lunt) - vvfat support - bugfix: close images on exit - SB16 - converted the sound output module stuff to the new 'soundmod' plugin - SF patches applied [3164945] hack to compile under WIN64 by Darek Mihocka and Stanislav [3164073] Fine grain SMC invalidation by Stanislav [1539417] write protect for floppy drives by Ben Lunt [2862322] fixes for emulated DHCP in eth_vnet - these S.F. bugs were closed/fixed [2588085] Mouse capture [3140332] typo in mf3/ps2 mapping of BX_KEY_CTRL_R [3111577] No "back" option in log settings [3108422] Timing window in NE2K emulation [3084390] Bochs won't load floppy plugin right on startup [3043174] Docbook use of '_' build failure [3085140] Ia_arpl_Ew_Rw definition of error [3078995] ROL/ROR/SHL/SHR modeling wrong when dest reg is 32 bit [2864794] BX_INSTR_OPCODE in "cpu_loop" causes crash in x86_64 host [2884071] [AIX host] prefetch: EIP [00010000] > CS.limit [0000ffff] [3053542] 64 bit mode: far-jmp instruction is error [3011112] error compile vs2008/2010 with X2APIC [3002017] compile error with vs 2010 [3009767] guest RFLAGS.IF blocks externel interrupt in VMX guest mode [2964655] VMX not enabled in MSR IA32_FEATURE_CONTROL [3005865] IDT show bug [3001637] CMOS MAP register meaning error [2994370] Cannot build with 3DNow support - these S.F. feature requests were closed/implemented [1510142] Native Windows XP x64 Edition binary [1062553] select mouse (de)activation in bochsrc [2930633] legacy mouse capture key : not specific enough [2930679] Let user change mouse capture control key [2803538] Show flags for pages when using "info tab" ------------------------------------------------------------------------- Changes in 2.4.5 (April 25, 2010): Brief summary : - Major configure/cpu rework allowing to enable/disable CPU options at runtime through .bochsrc (Stanislav) - Bugfixes for CPU emulation correctness and stability - Implemented X2APIC extensions (Stanislav) - Implemented Intel VMXx2 extensions (Stanislav) - Extended VMX capability MSRs, APIC Virtualization, X2APIC Virtualization, Extended Page Tables (EPT), VPID, Unrestricted Guests, new VMX controls. - Implemented PCLMULQDQ AES instruction - Extended Bochs internal debugger functionality - USB HP DeskJet 920C printer device emulation (Ben Lunt) Detailed change log : - Configure rework - Deprecate --enable-popcnt configure option. POPCNT instruction will be enabled automatically iff SSE4_2 is supported (like in hardware). - Make --ignore-bad-msrs runtime option in .bochsrc. Old --ignore-bad-msrs configure option is deprecated and should not be used anymore. - Enable changing part of CPU functionality at runtime through .bochsrc. - Now you could enable/disable any of SSEx/AES/MOVBE/SYSENTER_SYSEXIT/XSAVE instruction sets using new CPUID option in .bochsrc. - When x86-64 support is compiled in, you could enable/disable long mode 1G pages support without recompile using new CPUID option in .bochsrc. Configure options: --enable-mmx, --enable-sse, --enable-movbe, --enable-xsave, --enable-sep, --enable-aes, --enable-1g-pages are deprecated and should not be used anymore. - Local APIC configure option --enable-apic is deprecated and should not be used anymore. The LAPIC option now automatically determined from other configure options. XAPIC functionality could be enabled using new CPUID .bochsrc option. - Changed default CPU configuration (generated by configure script with default options) to BX_CPU_LEVEL=6 with SSE2 enabled. - CPU - Implemented PCLMULQDQ AES instruction - Implemented X2APIC extensions / enable extended topology CPUID leaf (0xb), in order to enable X2APIC configure with --enable-x2apic - Implemented Intel VMXx2 extensions: - Enabled extended VMX capability MSRs - Implemented VMX controls for loading/storing of MSR_PAT and MSR_EFER - Enabled/Implemented secondary proc-based vmexec controls: - Implemented APIC virtualization - Implemented Extended Page Tables (EPT) mode - Implemented Descriptor Table Access VMEXIT control - Implemented RDTSCP VMEXIT control - Implemented Virtualize X2APIC mode control - Implemented Virtual Process ID (VPID) - Implemented WBINVD VMEXIT control - Implemented Unrestricted Guest mode In order to enable emulation of VMXx2 extensions configure with --enable-vmx=2 option (x86-64 must be enabled) - Bugfixes for CPU emulation correctness - Fixed Bochs crash when accessing the first byte above emulated memory size - Internal Debugger - Introduced range read/write physical watchpoints - Allow reloading of segment registers from internal debugger - Improved verbose physical memory access tracing - BIOS - Fix MTRR configuration (prevented boot of modern Linux kernels) - Fix interrupt vectors for INT 60h-66h (reserved for user interrupt) by setting them to zero - Fix BIOS INT13 function 08 when the number of cylinders on the disk = 1 - I/O Devices - USB HP DeskJet 920C printer device emulation (Ben Lunt) - Misc - Updated Bochs TESTFORM to version 0.5 - SF patches applied [2864402] outstanding x2apic patches by Stanislav [2960379] Fix build with -Wformat -Werror=format-security by Per Oyvind Karlsen [2938273] allow instrumentation to change execute by Konrad Grochowski [2926072] Indirection operators in expressions by Derek Peschel [2914433] makesym.perl misses symbols by John R. Jackson [2908481] USB Printer by Ben Lunt - these S.F. bugs were closed/fixed [2861662] dbg_xlate_linear2phy needs to be updated [2956217] INT13 AH=8 returns wrong values when cylinders=1 [2981161] Allow DMA transfers to continue when CPU is in HALT state [2795115] NX fault could be missed [2964824] bad newline sequence in aspi-win32.h [913419] configure options and build process needs some work [2938398] gdbstub compile error with x86_64 enabled [2734455] shutdown/reset type 05 should reinit the PICs [1921294] extended memory less than 1M wrong size [1947249] BX_USE_EBDA_TABLES and MP table placement [1933859] BX_USE_EBDA_TABLES and memory overlapping [2923680] "help dregs" is a syntax error [2919661] CPU may fail to do 16bit near call [2790768] Memory corruption with SMP > 32, Panic BIOS Keyboard Error [2902118] interrupts vectors 0x60 to 67 should be NULL ! [2912502] Instruction Pointer behaving erratically [2901047] Bochs crashed, closed by guest os [2905385] Bochs crash [2901481] Instruction SYSRET and SS(PL) [2900632] Broken long mode RETF to outer priviledge with null SS [1429011] Use bx_phyaddr_t for physaddr vars and bx_adress for lin adr - these S.F. feature requests were closed/implemented [2955911] RPM preuninstall scriptlet removes /core [2947863] don't abort on unrecognised options [2878861] numerics in the disassembler output [2900619] make more CPU state changeable ------------------------------------------------------------------------- Changes in 2.4.2 (November 12, 2009): - CPU and internal debugger - VMX: Implemented TPR shadow VMEXIT - Bugfixes for CPU emulation correctness (mostly for VMX support). - Bugfixes and updates for Bochs internal debugger - On SMP system stepN command now affects only current processor - Memory - Bugfixes for > 32-bit physical address space. - Allow to emulate more physical memory than host actually could or would like to allocate. For more details look for new .bochsrc 'memory' option. - Cleanup configure options - All paging related options now will be automatically determined according to --enable-cpu-level option. Related configure options --enable-global-pages, --enable-large-pages, --enable-pae, --enable-mtrr are deprecated now. Only 1G paging option still remaining unchanged. - Deprecate --enable-daz configure option. Denormals-are-zeros MXCSR control will be enabled automatically iff SSE2 is supported (like in hardware). - Deprecate --enable-vme configure option, now it will be supported iff CPU_LEVEL >= 5 (like in hardware). - I/O Devices - Bugfixes for 8254 PIT, VGA, Cirrus-Logic SVGA, USB UCHI - SF patches applied [2817840] Make old_callback static by Mark Marshall [2874004] fix for VMWRITE instruction by Roberto Paleari [2873999] fix CS segment type during fast syscall invocation by Roberto Paleari [2864389] Debugger gui maximize on startup by Thomas Nilsen [2817868] Rework loops in the memory code by Mark Marshall [2812948] PIT bug by Derek - these S.F. bugs were closed/fixed [2833504] GUI debugger bug-about GDT display [2872244] BIOS writes not allowed value to MTRR MSR causing #GP [2885383] SDL GUI memory leak [2872290] compilation in AIX5.3 ML10 failes [2867904] crash with cirrus bx_vga_c::mem_write [2851495] BIOS PCI returns with INT flag = 0 [2860333] vista 64 guest STOP 109 (GDT modification) [2849745] disassembler bug for 3DNow and SSE opcodes [1066748] Wrong registers values after #RESET, #INIT [2836893] Regression: Windows XP installer unable to format harddrive [2812239] VMX: VM-Exit: Incorrect instruction length on software int [2814130] bx_debug lex/yacc files incorrectly generated [2813199] MP Tables Missing From BIOS [2824093] VMX exception bug [2811909] VMX : CS Access-rights Type.Accessed stays 0 [2810571] Compile Errors on OSX [2823749] GCC regression or VM_EXIT RDMSR/WRMSR bug [2815929] Vista/XP64 unnecessary panic [2803519] Wrong example in man page bochsrc - these S.F. feature requests were closed/implemented [422766] Large Memory configurations [1311287] Idea for a better GUI [455971] USB support [615363] debugger shortcut for repeat last cmd ------------------------------------------------------------------------- Changes in 2.4.1 (June 7, 2009): - Fixed bunch of CPUID issues - Bochs is now able to install and boot 64-bit Windows images! (special thanks to Mark Ebersole for his patch) - Several bugfixes in CPU emulation (mostly for x87 instructions) - Fixed two critical deadlock bugs in the Win32 gui (patches from @SF tracker) - Fixes related to the 'show ips' feature - removed conflicting win32-specific alarm() functions ('win32' and 'sdl' gui) - feature now works in wx on win32 - Added support for gdb stub on big endian machine (patch by Godmar Back) - Rewritten obsolete hash_map code in dbg symbols module (patch from @SF) - BIOS: implemented missing INT 15h/89h (patch by Sebastian Herbszt) ------------------------------------------------------------------------- Changes in 2.4 (May 3, 2009): Brief summary : - Added graphical Bochs debugger frontend for most of the supported platforms. - Thanks for Chourdakis Michael and Bruce Ewing. - Many new CPU features in emulation - Support for > 32 bit physical address space and configurable MSRs - VMX, 1G pages in long mode, MOVBE instruction - Bugfixes for CPU emulation correctness, debugger and CPU instrumentation. - New config interface 'win32config' with start and runtime menu - USB: added OHCI support, external hub and cdrom - Added user plugin interface support. Detailed change log : - CPU and internal debugger - Support for VMX hardware emulation in Bochs CPU, to enable configure with --enable-vmx option Nearly complete VMX implementation, with few exceptions: - Dual-monitor treatment of SMIs and SMM not implemented yet - NMI virtualization, APIC virtualization not implemented yet - VMENTER to not-active state not supported yet - No advanced features like Extended Page Tables or VPID - Support for configurable MSR registers emulation, to enable configure with --enable-configurable-msrs option Look for configuration example in .bochsrc and msrs.def - Support new Intel Atom(R) MOVBE instruction, to enable configure with --enable-movbe option - Support for 1G pages in long mode, to enable configure with --enable-1g-pages option - Support for > 32 bit physical address space in CPU. Up to 36 bit could be seen in legacy mode (PAE) and up to 40 bit in x86-64 mode. Still support the same amount of the physical memory in the memory object, so system with > 4Gb of RAM yet cannot be emulated. To enable configure with --enable-long-phy-address option. - Implemented modern BIOSes mode limiting max reported CPUID function to 3 using .bochsrc CPU option. The mode is required in order to correctly install and boot WinNT. - Added ability to configure CPUID vendor/brand strings through .bochsrc (patch from @SF by Doug Reed). - Many bugfixes for CPU emulation correctness (both x86 and x86-64). - Updated CPU instrumentation callbacks. - Fixed Bochs internal debugger breakpoints/watchpoints handling. - Configure and compile - Added ability to choose Bochs log file name and Bochs debugger log file name from Bochs command line (using new -log and -dbglog options) - Removed Peter Tattam's closed source external debugger interface from the code. - Removed --enable-guest2host-tlb configure option. The option is always enabled for any Bochs configuration. - Removed --enable-icache configure option. The option is always enabled for any Bochs configuration. Trace cache support still remains optional and could be configured off. - Added configure option to compile in GUI frontend for Bochs debugger, to enable configure with --enable-debugger-gui option. The GUI debugger frontend is enabled by default with Bochs debugger. - Removed --enable-port-e9-hack configure option. The feature now could be configured at runtime through .bochsrc. - Added configure option to enable/disable A20 pin support. Disabling the A20 pin support slightly speeds up the emulation. - reduced dependencies between source files for faster code generation - BIOS - Added S3 (suspend to RAM) ACPI state to BIOS (patch by Gleb Natapov) - Implemented MTRR support in the bios (patches by Avi Kivity and Alex Williamsion with additions by Sebastian Herbszt) - Bug fixes - I/O Devices - Added user plugin support - remaining devices converted to plugins: pit, ioapic, iodebug - added 'plugin_ctrl' bochsrc option to control the presence of optional device plugins without a separate option. By default all plugins are enabled. - added register mechanism for removable mouse and keyboard devices - Hard drive / cdrom - PACKET-DMA feature now supported by all ATAPI commands - ATAPI command 0x1A added (based on the Qemu implementation) - sb16 - Added ALSA sound support on Linux (PCM/MIDI output) - FM synthesizer now usable with MIDI output (simple piano only) - Fixed OPL frequency to MIDI note translation - Fixed MIDI output command - keyboard - added keyboard controller commands 0xCA and 0xCB - USB - USB code reorganized to support more HC types and devices - added USB OHCI support written by Ben Lunt - added external USB hub support (initial code ported from Qemu) - added USB cdrom support (SCSI layer ported from Qemu) - added status bar indicators to show data transfer - VGA - VBE video memory increased to 16 MB - implemented changeable VBE LFB base address (PCI only, requires latest BIOS and VGABIOS images) - I/O APIC - implemented I/O APIC device hardware reset - Config interface - new config interface 'win32config' with start and runtime menu is now the default on Windows ('textconfig' is still available) - win32 device config dialogs are now created dynamicly from a parameter list (works like the wx ParamDialog) - changes in textcofig and the wx ParamDialog for compatibility with the new win32 dialog behaviour - Bochs param tree index keys are case independent now - some other additions / bugfixes in the simulator interface code - Misc - updated LGPL'd VGABIOS to version 0.6c - Updated Bochs TESTFORM to version 0.4 - SF patches applied [2784858] IO Handler names are not compared properly [2712569] Legacy bios serial data buffer timeout bug by grybranix [2655090] 64 bit BSWAP with REX.W broken by M. Eby [2645919] CR8 bug when reading by M. Eby [1895665] kvm: bios: add support to memory above the pci hole by Izik Eidus [2403372] rombios: check for valid cdrom before using it by Sebastian [2307269] acpi: handle S3 by Sebastian [2354134] TAP networking on Solaris/Sparc repaired [2144692] The scsi device can not complete its writing data command by naiyue [1827082] [PATCH] Configurable CPU vendor by Marcel Sondaar [2217229] Panic on EBDA overflow in rombios32 by Sebastian [2210194] Log pci class code by Sebastian [1984662] red led for disk write and titlebar mod by ggbsf [2142955] Fix for monitor/mwait by Doug Gibson [2137774] Patch to fix bug: cdrom: read_block: lseek returned error by Gabor Olah [2134642] Fix scan_to_scanascii table for F11 and F12 by Ben Guthro & Steve Ofsthun [2123036] sdl fullscreen fix by ggbsf [2073039] Remove CMOS accsess from AML code by Gleb Natapov [2072168] smbios: add L1-L3 cache handle to processor information by Sebastian [2055416] bochsrc cpu options for cpuid vendor and brand string by Doug Reed [2035278] rombios: Fix return from BEV via retf by Sebastian [2035260] rombios: El Torito load segment fix by Sebastian [2031978] Fix VMware backdoor command 0Ah by Jamie Lokier [2015277] Remove obsolete comment about DATA_SEG_DEFS_HERE hack by Sebastian [2011268] Set new default format and unit only if both are supported by Sebastian [2001919] gdbstub: fix qSupported reply by Sebastian [2001912] gdbstub: enclose packet data by apostrophes by Sebastian [1998071] fix missing SIGHUP and SIGQUIT with term ui on mingw by Sebastian [1998063] fix wrong colors with term ui by Sebastian [1995064] Compile fix needed for --enable-debugger and gcc 4.3 by Hans de Goede [1994564] Fix typo in RDMSR BX_MSR_MTRRFIX16K_A0000 by Sebastian [1994396] Change hard_drive_post #if by Sebastian [1993235] TESTFORM email address update by Sebastian [1992322] PATCH: fix compilation of bochs 2.3.7 on bigendian machines by Hans de Goede [1991280] Shutdown status code 0Ch handler by Sebastian [1990108] Shutdown status code 0Bh handler by Sebastian [1988907] Shutdown status code 0Ah handler by Sebastian [1984467] two typos in a release! (2.3.7) [1981505] Init PIIX4 PCI to ISA bridge and IDE by Sebastian - these S.F. bugs were closed/fixed [2784148] an integer overflow BUG of Bochs-2.3.7 source code [2695273] MSVC cpu.dsp failure in 2.3.7.zip [616114] Snapshot/Copy crash on Win2K [2628318] 'VGABIOS-latest' bug [1945055] can't 'make install' lastest bochs on loepard [2031993] Mac OS X Makefile bug [1843199] install error on mac osx [2710931] Problem compiling both instrumentation and debugger [2617003] ExceptionInfo conflicts with OS X api [2609432] stepping causes segfault (CVS) [2605861] compile error with --enable-smp [1757068] current cvs(Jul19, 07) failed to boot smp [2426271] cannot get correct symbol entry [2471982] VGA character height glitches [1659659] wrong behaviour a20 at boot [1998027] minwg + --with-term + --with-out-win32 = link failure [1871936] bochs-2.3.6 make fails on wx.cc [1684666] info idt for long mode [2105989] could not read() hard drive image file at byte 269824 [1173093] Debugger totally not supports x86-64 [1803018] new win32debug dialog problems [2141679] windows vcc build broken [2162824] latest cvs fails to compile [2164506] latest bochs fails to start [2129223] MOV reg16, SS not working in real mode due to dead code [2106514] RIS / startrom.com install ALMOST works [2123358] SMP (HTT): wbinvd executed by CPU1 crashes CPU0 [2002758] Arch Linux: >>PANIC<< ATAPI command with zero byte count [2026501] El Torito incorrect boot segment:offset [2029758] BEV can return via retf instead of int 18h [2010173] x command breaks after one error about x/s or x/i [1830665] harddrv PANIC: ATAPI command with zero byte count [1985387] fail to make using gcc4 with --enable-debugger [1990187] testform feedback [1992138] Misspell in cpu/ia_opcodes.h - these S.F. feature requests were closed/implemented [2175153] Update MSVC project files [658800] front end program and bios [1883370] Make cd and floppy images more usable [422783] change floppy size without restarting [2552685] param tree names should be case insensitive [1214659] PC Speaker emu turnoff. Plugin Controll. [1977045] support 40 bit physical address [1506385] Intel Core Duo VT features [1429015] Support for user plugins [1488136] debugger access to floppy controller [1363136] Full debugger SMP and 64 bit support [2068304] Support for ACPI [431032] debugger "x" command [423420] profiling ideas (SMF) [445342] Add FM support? [928439] alsa ------------------------------------------------------------------------- Changes in 2.3.7 (June 3, 2008): Brief summary : + More optimizations in CPU code - Bochs 2.3.7 is more than 2x faster than Bochs 2.3.5 build ! - Implemented LBA48 support in BIOS - Added memory access tracing for Bochs internal debugger - Implemented Intel(R) XSAVE/XRSTOR and AES instruction set extensions - Many fixes in CPU emulation and internal debugger - MenuetOS64 floppy images booting perfect again ! - updated LGPL'd VGABIOS to version 0.6b Detailed change log : - CPU - Support of XSAVE/XRSTOR CPU extensions, to enable configure with --enable-xsave option - Support of AES CPU extensions, to enable configure with --enable-aes option - Fixed Bochs failure on RISC host machines with BxRepeatSpeedups optimization enabled - Implemented SYSENTER/SYSEXIT instructions in long mode - More than 100 bugfixes for CPU emulation correctness (both x86 and x86-64) - MenuetOS64 floppy images booting perfect again ! - Updated CPU instrumentation callbacks - Bochs Internal Debugger and Disassembler - Added memory access tracing for Bochs internal debugger, enable by typing 'trace-mem on' in debugger command line - Many bug fixes in Bochs internal debugger and disassembler - System BIOS (Volker) - Implemented LBA48 support - Added generation of SSDT ACPI table that contains definitions for available processors - Added RTC device to ACPI DSDT table - Added implementation of SMBIOS - I/O devices (Volker) - VGA - Implemented screen disable bit in sequencer register #1 - Implemented text mode cursor blinking - Serial - new serial modes 'pipe-server' and 'pipe-client' for win32 - new serial mode 'socket-server' - Configure and compile - Fixed configure bug with enabling of POPCNT instruction, POPCNT instruction should be enabled by default when SSE4.2 is enabled. - Removed --enable-magic-breakpoint configure option. The option is automatically enabled if Bochs internal debugger is compiled in. It is still possible to turn on/off the feature through .bochsrc. - Allow boot from network option in .bochsrc - Added Bochs version info for Win32 - Display libraries - implemented text mode character blinking in some guis - improved 'X' gui runtime dialogs - SF patches applied [1980833] Fix shutdown status code 5h handler by Kevin O'Connor [1928848] "pipe" mode for serial port (win32 only) by Eugene Toder [1956843] Set the compatible pci interrupt router back to PIIX by Sebastian [1956366] Do not announce C2 & C3 cpu power state support by Igor Lvovsky [1921733] support for LBA48 by Robert Millan [1938185] Fix link problem with --enable-debugger by Sebastian [1938182] Makefile.in - use @IODEV_LIB_VAR@ by Sebastian [1928945] fix for legacy rombios - e820 map and ACPI_DATA_SIZE by Sebastian [1925578] rombios32.c - fix ram_size in ram_probe for low memory setup by Sebastian [1908921] rombios32.c - move uuid_probe() call by Sebastian [1928902] improvements to load-symbols by Eugene Toder [1925568] PATCH: msvc compilation by Eugene Toder [1913150] rombios.c - e820 cover full size if memory <= 16 mb by Alexander van Heukelum [1919804] rombios.c - fix and add #ifdef comments by Sebastian [1909782] rombios.c - remove segment values from comment by Sebastian [1908918] SMBIOS - BIOS characteristics fix by Sebastian [1901027] BIOS boot menu support (take 3) [1902579] rombios32.c - define pci ids by Sebastian [1859447] Pass segment:offset to put_str and introduce %S by Sebastian [1889057] rombios.c - boot failure message by Sebastian [1891469] rombios.c - print BEV product string by Sebastian [1889851] Win32 version information FILEVERSION for bochs.exe by Sebastian [1889042] rombios.c - fix comment by Sebastian [1881500] bochsrc, allow boot: network by Sebastian [1880755] Win32 version information for bochs.exe by Sebastian [1880471] SMBIOS fix type 0 by Sebastian [1878558] SMBIOS fixes by Sebastian [1864692] SMBIOS support by Filip Navara [1865105] Move bios_table_area_end to 0xcc00 by Sebastian [1875414] Makefile.in - change make use by Sebastian [1874276] Added instrumentation for sysenter/sysexit by Lluis [1873221] TLB page flush: add logical address to instrumentation by Lluis [1830626] lba32 support by Samuel Thibault [1861839] Move option rom scan after floppy and hard drive post by Sebastian [1838283] Early vga bios init by Sebastian [1838272] rom_scan range parameter by Sebastian [1864680] Save CPUID signature by Filip Navara - these S.F. bugs were closed [1976171] Keyboard missing break code for enter (0x9C) [666433] physical read/write breakpoint sometimes fails [1744820] info gdt and info idt shows the entire tables [1755652] graphics: MenuetOS64 shows black screen [1782207] Windows Installer malfunction, Host=Linux, Guest=Win98SE [1697762] OS/2 Warp Install Failed [1952548] String to char * warnings [1940714] SYSENTER/SYSEXIT doesn't work in long mode [1422342] SYSRET errors [1923803] legacy rombios - e820 map and ACPI_DATA_SIZE [1936132] Link problem with --enable-debugger & --enable-disasm [1934477] Linear address wrap is not working [1424984] virtual machine freezes in Bochs 2.2.6 [1902928] with debugger cpu_loop leaves CPU with unstable state [1898929] Bochs VESA BIOS violates specs (banks == 1) [1569256] bug in datasegment change in long mode [1830662] ACPI: no DMI BIOS year, acpi=force is required [1868806] VGA blink enable & screen disable [1875721] Bit "Accessed" in LDT/GDT descriptors & #PF [1874124] bx_Instruction_c::ilen() const [1873488] bochs-2.3.6 make fails on dbg_main.cc - these S.F. feature requests were implemented [1422769] SYSENTER/SYSEXIT support in x86-64 mode [1847955] Version information for bochs(dbg).exe [939797] SMBIOS support ------------------------------------------------------------------------- Changes in 2.3.6 (December 24, 2007): Brief summary : + More than 25% emulation speedup vs Bochs 2.3.5 release! - Thanks to Darek Mihocka (http://www.emulators.com) for providing patches and ideas that made the speedup possible! + Up to 40% speedup vs Bochs 2.3.5 release with trace cache optimization! - Lots of bugfixes in CPU emulation - Bochs benchmarking support - Added emulation of Intel SSE4.2 instruction set Detailed change log : - CPU - Added emulation of SSE4.2 instruction set, to enable use --enable-sse=4 --enable-sse-extension configure options to enable POPCNT instruction only use configure option --enable-popcnt - Implemented MTRR emulation, to enable use --enable-mtrr configure option. MTRRs is enabled by default when cpu-level >= 6. - Implemented experimental MONITOR/MWAIT support including optimized MWAIT CPU state and hardware monitoring of physical address range, to enable use --enable-monitor-mwait configure option. - Removed hostasm optimizations, after Bochs rebenchmarking it was found that the feature bringing no speedup or even sometimes slows down emulation! - Merged trace cache optimization patch, the trace cache optimization is enabled by default when configure with --enable-all-optimizations option, to disable trace cache optimization configure with --disable-trace-cache - Many minor bugfixes in CPU emulation (both ia32 and x86-64) - Updated CPU instrumentation callbacks - Bochs Internal Debugger and Disassembler - Many fixes in Bochs internal debugger and disassembler, some debugger interfaces significantly changed due transition to the param tree architecture - Added support for restoring of the CPU state from external file directly from Bochs debugger - Configure and compile - Renamed configure option --enable-4meg-pages to --enable-large-pages. The option enables page size extensions (PSE) which refers to 2M pages as well. - Removed --enable-save-restore configure option, save/restore feature changed to be one of the basic Bochs features and compiled by default for all configurations. - Added new Bochs benchmark mode. To run Bochs in benchmark mode execute it with new command line option 'bochs -benchmark time'. The emulation will be automatically stopped after 'time' millions of emulation cycles executed. - Another very useful option for benchmarking of Bochs could be enabled using new 'print_timestamps' directive from .bochsrc: print_timestamps: enable=1 - Added --enable-show-ips option to all configuration scripts used to build release binaries, so all future releases will enjoy IPS display. - Enable alignment check in the CPU and #AC exception by default for --cpu-level >= 4 (like in real hardware) - SF patches applied [1491207] Trace Cache Speedup patch by Stanislav [1857149] Define some IPL values by Sebastian [1850183] Get memory access mode in BX_INSTR_LIN_READ by Lluis Vilanova [1841421] pic: keep slave_pic.INT and master_pic.IRQ_in bit 2 in sync by Russ Cox [1841420] give segment numbers in exception logs by Russ Cox [1801696] Allow Intel builds on Mac OS X [1830658] Fix >32GB disk banner by Samuel Thibault [1813314] Move #define IPL_* and typedef ipl_entry by Sebastian [1809001] Save PnP Option ROM Product Name string in IPL Boot Table by Sebastian [1821242] Fix for #1801285, Niclist.exe broken by Sebastian [1819567] Code warning cleanup [1816162] Update comment on bios_printf() by Sebastian [1811139] Trivial Fix when BX_PCIBIOS and BX_ROMBIOS32 not defined by Myles Watson [1811190] Improve HD recognition and CD boot by Myles Watson [1811860] Implement %X in bios_printf by Sebastian [1809649] printf %lx %ld %lu by Myles Watson [1809651] move BX_SUPPORT_FLOPPY by Myles Watson [1809652] dpte and Int13DPT fixes by Myles Watson [1809669] clip cylinders to 16383 in hard drive by Myles Watson [1799903] Build BIOS on amd64 by Robert Millan [1799877] Fix for parallel build (make -j2) by Robert Millan - these S.F. bugs were closed [1837354] website bug: View the Source link broken [1801268] Reset from real mode no longer working [1843250] Using forward slashes gives invalid filename [1823446] BIOS bug, local APIC #0 not detected [1801285] Niclist.exe broken [1364472] breakpoints sometimes don't work [994451] breakpoint bug [1801295] NSIS installer vs Windows Notepad [1715328] Unreal mode quirk [1503972] debugger doesn't debug first instruction on exception [1069071] div al, byte ptr [ds:0x7c18] fails to execute [1800080] Wrong "BX_MAX_SMP_THREADS_SUPPORTED" assertion - these S.F. feature requests were implemented [1662687] Download for Win32-exe with x64 Mode and debugging [604221] Debugger command: query lin->phys mapping ------------------------------------------------------------------------- Changes in 2.3.5 (September 16, 2007): Brief summary : - Critical problems fixed for x86-64 support in CPU and Bochs internal debugger - ACPI support - The release compiled with x86-64 and ACPI - Hard disk emulation supports ATA-6 (LBA48 addressing, UDMA modes) - Added emulation of Intel SSE4.1 instruction set Detailed change log : - CPU - Fixed critical bug with 0x90 opcode (NOP) handling in x86-64 mode - implied stack references where the stack address is not in canonical form should causes a stack exception (#SS) - Added emulation of SSE4.1 instruction set (Stanislav) - Do not save and restore XMM8-XMM15 registers when not in x86-64 mode - Fixed zero upper 32-bit part of GPR in x86-64 mode - CMOV_GdEd should zero upper 32-bit part of GPR register even if the 'cmov' condition was false ! - Implemented CLFLUSH instruction, report non-zero cache size in CPUID - Fixed PUSHA/POPA instructions behavior in real mode - Fixed detection of inexact result by FPU - Fixed denormals-are-zero (DAZ) handling by SSE convert instructions - Implemented Misaligned Exception Mask support for SSE (MXCSR[17]) - Implemented Alignment Check in the CPU and #AC exception, to enable use --enable-alignment-check configure option - General - 2nd simulation support in wxBochs now almost usable (simulation cleanup code added and memory leaks fixed) - Configure and compile - several fixes for MacOSX, OpenBSD and Solaris 10 - enable save/restore feature by default for all configurations - reorganized SSE configure options to match Intel(R) Programming Reference Manual, new option introduced for SSE extensions enabling. To enable Intel Core Duo 2 new instructions use --enable-sse=3 --enable-sse-extension enabling of SSE4.1 (--enable-sse=4) will enable SSE3 extensions as well - removed old PIT, always use new PIT written by Greg Alexander, removed configure option --enable-new-pit - I/O devices (Volker) - Floppy - partial non-DMA mode support (patch by John Comeau) - Hard drive / cdrom - hard disk emulation now supports ATA-6 (LBA48 addressing, UDMA modes) - VMWare version 4 disk image support added (patch by Sharvil Nanavati) - PCI - initial support for the PIIX4 ACPI controller - Serial - added support for 3-button mouse with Mousesystems protocol - USB - experimental USB device change support added - rewrite of the existing USB devices code - new USB devices 'disk' and 'tablet' (ported from the Qemu project) - Bochs internal debugger - fixed broken debugger "rc file" option (execute debugger command from file) - implementation of a gui frontend ("windebug") for win32 started - gdbstub now accepts connection from any host - several documentation updates - a lot of disasm and internal debugger x86_64 support fixes - Configuration interface - fixes and improvements to the save state dialog handling - Display libraries - text mode color handling improved in some guis - win32 fullscreen mode (patch by John Comeau) - System BIOS (Volker) - 32-bit PM BIOS init code for ACPI, PCI, SMP and SMM (initial patches by Fabrice Bellard) - PCI BIOS function "find class code" implemented - SF patches applied [1791000] 15h 8600h is reading the wrong stack frame by Sebastian [1791016] rombios32.c, ram_probe(), BX_INFO missing value by Sebastian [1786429] typo in bochsrc.5 by Sebastian [1785204] Extend acpi_build_table_header to accept a revision number by Sebastian [1766536] Partial Patch for Bug Report 1549873 by Ben Lunt [1763578] ACPI Table Revision 0 -> 1 [1642490] implement alignment check and #AC exception by Stanislav Shwartsman [1695652] [PATCH] .pcap pktlog and vnet PXE boot by Duane Voth [1741153] Add expansion-ROM boot support to the ROMBIOS [1734159] Implemented INT15h, fn 0xC2 (mouse), subfn 3, set resolution [1712970] bios_printf %s fix [1573297] PUSHA/POPA real mode fix by Stanislav Shwartsman [1641816] partial support for non-DMA access to floppy by John Comeau [1624032] shows where write outside of memory occurred by John Comeau [1607793] allow fullscreen when app requests it by John Comeau [1603013] Bugfix for major NOP problem on x64 by mvysin [1600178] Make tap and tuntap compile on OpenBSD by Jonathan Gray [1149659] improve gdbstub network efficiency by Avi Kivity [1554502] Trivial FPU exception handling fix - these S.F. bugs were closed [1316008] Double faults when it shouldn't - gcc 4.0.2 [1787289] broken ABI for redolog class when enable-compressed-hd [1787500] tftp_send_optack not 64bit clean [1264540] Security issue with Bochs website [1767217] Debugger Faults including ud2 [1729822] Various security issues in io device emulation [1675202] mptable hosed (bad entry count in header) [1197141] 'make install' installs to bad location [1157623] x86Solaris10 cannot recoginize ACPI RSD PTR [1768254] large HDD in Bochs/bximage [1496157] Windows Vista Beta2 dosn't boot [1755915] Illegal Hard Disk Signature Output [1717790] info gdt and info idt scrolls away, too long result [1726640] Debugger displays incorrect segment for mov instruction [1719156] Typo in misc_mem.cpp [1715270] Debugger broken in/beyond 2.3 [1689107] v8086 mode priviledge check failed [1704484] A few checks when CPU_LEVEL < 4 [1678395] Problem with zero sector... [876990] SA-RTL OS fails on PIC configuration [1673582] save/restore didn't restore simulation correctly [1586662] EDD int 13h bug, modify eax [666618] POP_A Panic in DOS EMU [1001485] panic: not enough bytes on stack [1667336] delay times an order of magnitude slow [1665601] crash disassembling bootcode [1657065] CVS sources won't compile [1653805] bochs's gdbstub uses incorrect protocol [1640737] ASM sti command frezzes guest OS [1636439] latest CVS sources don't compile under Cygwin [1634357] disasm incorrect (no sign ext) displacement in 64-bit mode [1376453] pcidev segfaults bochs [1180890] IOAPIC in BOCHS - WinXP 64 in MP version [1597528] 2.3 fails to compile on amd64 [1526255] FLD1 broken when compaling with gcc 4.0.x [1597451] eth_fbsd is broken under FreeBSD [1571949] Bochs will not compile under Solaris [1500216] Bochs fails to boot BeOs CD [1458339] bochs-2.2.6 WinXP Binary ACPI error installing FreeBSD 6.0 [1440011] patches needed for FreeBSD 6.0 to compile Bochs [431674] some devices don't have a prefix [458150] QNX demo disk crashes with new pit [818322] Bochs 2.1 cvs: OS/2 - read verify on non disk [906840] KBD: bogus scan codes generated in set 3 [1005053] No keyboard codes translation [1109374] Problem with Scancodeset 2 [1572345] Bochs won't continue [1568153] Bochs looks for (and loads?) unspecified display libraries [1563462] Errors in /iodev/harddrv.h [1562172] TLB_init() fails to initialize priv_check array if USE_TLB 0 [1385303] debugger crashes after panic [1438227] crc.cpp missing in bx_debug version 2.2.6 [1501825] debugger crashes on to high input [1420959] Memory leak + buffer overflow in Bochs debugger [1553289] Error in Dis-assembler [542464] I cannot use FLAT [1548270] Bochs won't die with its pseudo terminal [1545588] roundAndPackFloatx80 does not detect round up correctly ------------------------------------------------------------------------- Changes in 2.3 (August 27, 2006): Brief summary : - limited save/restore support added (config + log options, hardware state) - configuration parameter handling rewritten to a parameter tree - lots of cpu and internal debugger fixes - hard disk geometry autodetection now supported by most of the image types - hard disk emulation now supports ATA-3 (multiple sector transfers) - VBE memory size increased to 8MB and several VGA/VBE fixes - updated LGPL'd VGABIOS to version 0.6a Detailed change log : - CPU and internal debugger fixes - Fixed bug in FSTENV instruction (Stanislav Shwartsman) - Recognize #XF exception (19) when SSE is enabled - Fixed bug in PSRAW/PSRAD MMX and SSE instructions - Save and restore RIP/RSP only for FAULT-type exceptions, not for traps - Correctly decode, disassemble and execute multi-byte NOP '0F F1' opcode - Raise A20 line after system reset (Stanislav Shwartsman) - Implemented SMI and NMI delivery (APIC) and handling in CPU (Stanislav) - Experimental implementation of System Management Mode (Stanislav) - Added emulation of SSE3E instructions (Stanislav Shwarstman) - Save and restore FPU opcode, FIP and FDP in FXSAVE/FRSTOR instructions - Fixed bug in MOVD_EdVd opcode (always generated #UD exception) - Fixed critical issue, Bochs was not supporting > 16 bit LDT.LIMIT values - Many fixes in Bochs internal debugger and disassembler - CPU x86-64 fixes - Fixed SYSRET instruction implementation - Fixed bug in CALL/JMP far through 64-bit callgate in x86-64 mode - Correctly decode, disassemble and execute 'XCHG R8, rAX' instruction - Correctly decode and execute 'BSWAP R8-R15' instructions - Fixed ENTER and LEAVE instructions in x86-64 mode (Stanislav) - Fixed CR4 exception condition (No Name) - Fixed x86 debugger to support x86-64 mode (Stanislav) - APIC and SMP - Support for Dual Core and Intel(R) HyperThreading Technology. Now you could choose amount of cores per processor and amount of HT threads per core from .bochsrc for SMP simulation (Stanislav Shwartsman) - Allow to control SMP quantum value through .bochsrc CPU option parameter. Previous Bochs versions used hardcoded quantum=5 value. - Fixed interrupt priority bug in service_local_apic() - Fixed again reading of APIC IRR/ISR/TMR registers. Finally it becomes fully correct :-) - Configure and compile - Moved configure time --enable-reset-on-triple-fault option to runtime, the 'cpu' option in .bochsrc is extended and the old configure option is deprecated (Stanislav Shwartsman) - Removed --enable-pni configure option, to compile with PNI use --enable-sse=3 instead (Stanislav Shwartsman) - enable SEP (SYSENTER/SYSEXIT) support by default for Penitum II+ processor emulation (i.e. if cpu-level >= 6 and MMX is enabled) - general - Limited save/restore support added. The state of CPU, memory and all devices can be saved now (state of harddisk images not handled yet). - Fixed several memory leaks - configuration interface - Configuration parameter handling rewritten to a parameter tree. This is required for dynamic menus/dialogs, user-defined options and save/restore. - Support for user-defined bochsrc options added - help support at the parameter prompt in textconfig added - I/O devices (Volker) - Floppy - partial sector transfers fixed - Hard drive / cdrom - several fixes to the IDE register behaviour (e.g. in case of a channel with only one drive connected) - fixed data alignment of 'growing' hard drive images (sharing images between Windows and Linux now possible) - disk geometry autodetection now supported by most of the image types (unsupported: external, dll and compressed modes) - multi sector read/write commands implemented - hard disk now reporting ATA-3 supported - ATAPI 'inquiry' now returns a unique device name - Keyboard - reset sent to keyboard has no effect on the 8042 (scancode translation) - PCI - forward PIRQ register changes to the I/O APIC (if present) - attempt to fix and update the emulation part of 'pcidev' (untested) - VGA - VBE memory size increased to 8MB and several VBE fixes - VGA memory read access fixed (bit plane access and read mode) - VGA memory is now a part of the common video memory - System BIOS (Volker) - enable interrupts before executing INT 19h - fixed ATA device detection in case of one drive only connected to controller - improved INT 15h function AX=E820h - real mode PCI BIOS now returns IRQ routing information (function 0Eh) - keyboard LED flags handling fixed and improved - fixed handling of extended keys in INT 09h - Updated LGPL'd VGABIOS to version 0.6a - SF patches applied [1340111] fixes and updates to usb support by Ben Lunt [1539420] minor addition to pci_usb code by Ben Lunt [1455958] call/jmp through call gate in 64-bit mode [1433107] PATCH: fix compile with wxwindows 2.6 (unicode / utf8) by jwrdegoede [1386671] Combined dual core and hyper-threading patch - these S.F. bugs were closed [833927] TTD: System Error TNT.40025: Unexpected processor exception [789230] Sending code that shows lock up when setting idt [909670] Problems with Symantec Ghost [1540241] include missing in osdep.cc [1539373] Incorrect disasm for "mov moffset,bla" in 64bit [1538419] incorrect disassembly of [rip+disp] with rex.b [1535432] shift+cursor key maps to a digit [1504891] Knoopix 5.0.1 error [1424355] bochs-2.2.6 ata failure in windoze 98se [1533979] wrong disassembly of IN instruction [620059] paste won't stop [1164904] status bar doesn't show num/caps/scroll lock status [1061720] ATA Support level for HD [1522196] Broken CHANGES link in main page [1438415] crash if screen scrolled downwards [778441] Shouldn't interrupts be enable after BIOS? [1514949] I got a problem with the 8253 timer [1513544] disasm of 0xec (in AL,DX) returns ilen of 2 instead of 1 [1508947] APIC interrupt priority checking and interrupt delivery [766286] Debugger halts after any GPF exception [639143] va_list is not a pointer on linuxppc [1501815] debugger examines memory over page-boundary wrong [1503978] movsb/w/d doesn't work when direction is stored [1499405] WinPCap has changed URL hosting [1498519] APIC IRR bits not set while interrupts disabled [1498193] Bochs segfaults on LTR instruction [787140] Guest2HostTLB optimization bug [1492070] instrument stop [1487772] No SEP on P4 [1488335] Growing hard disk images severe interoperability errors! [1076312] Shadow RAM and TLB [1282249] The real i440FX chipset Award bios hangs [1479763] mistake "mov ax,[es:di]" for "mov ax,[ds:di]" [1453575] Misconfigured floppy DMA transfers do not terminate. [1460068] Incorrect handling for the Options Menu Item [910203] bochs-2.1.1 wx.lo failed [1438654] PANIC when trying to run install-amd64-minimal-2005.0.iso [1458320] compile hdimage.h fails [1455880] bochs-2.2.6,2: make error on FreeBSD [696890] Network wouldn't run under W2k hosting MSDOS [673391] SMP timer problems [1291059] wxWindows GUI on non-windows/configure issue [1356450] bochs 2.2.1 errors-omittions [1178017] Win98 guest cannot receive network packets from host [1076315] a20_mask after restarting [1436323] real hw does not panic when bad Ib in CMPSS_VssWssIb [1435269] cdrom_amigaos is not compilable [1433314] disasm issues [1170614] relative jumps/calls wrong in debugger [758121] user might get confused when interrupt handler invoked [1170622] You cannot toggle OFF "show" flags [1406387] JMP instruction should display absolute address [1428813] PANIC: ROM address space out of range [1426288] DR-DOSs EMM386 problem [1412036] Bochs cannot recognize PCI NIC correctly [435115] dbg: modebp broken and no docs [1419366] disasm cs:eip does not work anymore [1419393] SSE's #XF exception -> "exception(19): bad vector" [1419429] disassembly of "260f6f00" show DS: instead of ES: prefix [1417583] Interrupt behaviour changed from 2.2.1 to 2.2.5 [1418281] 'push' (6A) incorrectly disassembled [1417791] FLDENV generating exception when real hw does not. [1264583] OS/2 1.1 doesn't run ------------------------------------------------------------------------- Changes in 2.2.6 (January 29, 2006): - First major SMP release ! - several APIC and I/O APIC fixes make SMP Bochs booting Windows NT4.0 or Knoppix 4.0.2 without noapic kernel option in SMP configuration. - critical APIC timer bug fixed - obsolete SMP BIOS images removed (MP tables created dynamicaly) - determine number of processors in SMP configuration through .bochsrc new .bochsrc option 'CPU' allows to choose number of processors to emulate - new configure option --enable-smp to configure Bochs for SMP support, the old --enable-processors=N option is deprecated - CPU and internal debugger fixes - enabled #PCE bit in CR4 register, previosly setting of this bit generated #GP(0) fault - enabled LAHF/SAHF instructions in x86-64 mode - fixed bug in PMULUDQ SSE2 instruction - fixes in Bochs debugger - Configure and compile - enable VME (virtual 8086 mode extensions) by default if cpu-level >= 5 - enable Bochs disassembler by default for all configurations - win32 installer script improvements - ips parameter moved to new 'CPU' option - show IPS value in status bar if BX_SHOW_IPS is enabled - Other - several fixes in the hard drive, keyboard, timer, usb and vga code - new user button shortcut "bksl" (backslash) - updated Bochs instrumentation examples - user and development documentation improved ------------------------------------------------------------------------- Changes in 2.2.5 (December 30, 2005): Brief summary : - added virtual 8086 mode extensions (VME) implementation - several fixes/improvements in x86-64 emulation, debugger and disassembler - new serial mode 'socket' connects a network socket - IDE busmaster DMA feature for harddisks and cdroms completed and enabled - many improvements in Bochs emulated I/O devices (e.g. floppy, cdrom) - Updated LGPL'd VGABIOS to version 0.5d Detailed change log : - CPU - fixed XMM registers restore in FXRSTOR instruction (Andrej Palkovsky) - print registers dump to the log if tripple fault occured - fixed PANIC in LTR instruction (Stanislav) - added virtual 8086 mode extensions (VME) implementation, to enable configure with --enable-vme (Stanislav) - flush caches and TLBs when executing WBINVD and INVD instructions - do not modify segment limit and AR bytes when modifying segment register in real mode (support for unreal mode) - fixed init/reset values for LDTR and TR registers - reimplemented hardware task switching mechanism (Stanislav) - generate #GP(0) when fetching instruction cross segment boundary - CPU (x86-64) (Stanislav Shwartsman) - implemented call_far/ret_far/jmp_far instructions in long mode - fixed IRET operation in long mode - fixed bug prevented setting of NXE/FFXSR bits in MSR.EFER register - implemented RDTSCP instruction - do not check CS.limit when prefetching instructions in long mode - fixed masked write instructions (MASKMOVQ/MASKMOVDQU) in long mode - fetchdecode fixes for x86-64 - APIC - Fixed bug in changing local APIC id (Stanislav) - Fixed reading of IRR/ISR/TMR registers (patch by wmrieker) - Implemented spurious interrupt register (Stanislav, patch by wmrieker) - Fixed interrupt delivery bug (anonymous #SF patch) - Correctly implemented ESR APIC register (Stanislav) - Bochs debugger - Fixed bug in bochs debugger caused breakpoints doesn't fire sometimes (Alexander Krisak) - watchpoints in device memory fixed (Nickolai Zeldovich) - new debug interface to access Bochs CPU general purpose registers with support for x86-64 - Disassembler (Stanislav Shwartsman) - Fixed disassembly for FCOMI/FUCOMI instructions - Full x86-64 support in disassembler. The disassembler module extended to support x86-64 extensions. Still limited by Bochs debugger which is not supporting x86-64 at all ;( - I/O devices (Volker) - general - memory management prepared for large BIOS images (up to 512k) - slowdown timer sleep rate fixed (now using 1 msec on all platforms) - some device specific parameter handlers moved into the device code - serial - new serial mode 'socket' connects a network socket (#SF patch by Andrew Backer) - hard drive / cdrom - assign a unique serial number to each drive (fixes harddrive detection problems with Linux kernels 2.6.x: "ignoring undecoded slave") - geometry autodetection for 'flat' hard disk images added. Works with images created with bximage (heads = 16, sectors per track = 63) - ATAPI command 'read cd' implemented, some other commands improved - cdrom read block function now tries up to 3 times before giving up - emulation of raw cdrom reads added, some other lowlevel cdrom fixes - IDE busmaster DMA feature for harddisks and cdroms completed and enabled - disk image size limit changed from 32 to 127 GB - split ATA/ATAPI emulation code and image handling code - floppy - fixes for OS/2 (patch by Robin Kay) - disk change line behaviour fixed (initial patch by Ben Lunt) - end-of-track (EOT) condition handling implemented - more accurate timing for read/write data and format track commands using a motor speed of 300 RPM - timing of recalibrate and seek commands now depends on the step rate, date rate and the steps to do - floppy controller type changed to 82077AA - cmos - RTC 12-hour and binary mode implemented - number of CMOS registers changed from 64 to 128 - bochsrc option 'cmosimage' improved - save cmos image on exit if enabled - speaker - simple speaker support for OS X added (patch by brianonn@telus.net) - pci - BeOS boot failure fix in the PCI IDE code - don't register i/o and memory regions during PCI probe - vga - memory allocation for vga extensions fixed - usb - some bugfixes by Ben Lunt (mouse and keypad are usable now) - networking modules - VDE networking module now enabled on Linux - display libraries - general - new syntax for the userbutton shortcut string and more keys supported - win32 - fixed keycode generation for right alt/ctrl/shift keys - runtime dialog is now a property sheet - x11 - simple dialog boxes for the "ask" and "user shortcut" feature implemented - Slovenian keymap added (contributed by Mitja Ursic) - configuration interface - ask dialog is now enabled by default for win32, wx and x display libraries - bochsrc option floppy_command_delay is obsolete now (floppy timing now based on hardware specs) - floppy image size detection now available in the whole config interface - some device specific parameter handlers moved into the device code - calculate BIOS ROM start address from image if not specified - System BIOS (Volker) - PCI i/o and memory base address initialization added - several keyboard interrupt handler fixes (e.g. patch by japheth) - several floppy fixes (e.g. OS/2 works with patch by Robin Kay) - some more APM functions added - Updated LGPL'd VGABIOS to version 0.5d - generate SMP specific tables dynamicly by the Bochs memory init code - SF patches applied [1389776] Disk sizes over 64 Gbytes by Andrzej Zaborowski [1359162] disasm support for x86-64 by Stanislav Shwartsman [857235] task priority and other APIC bugs, etc by wmrieker [1359011] build breaks for 386 + debugger + disasm by shirokuma [1352761] Infinite loop when trying to debug a triple exception [1311170] small APIC bug fix (interrupt sent to the wrong CPU) [1309763] Watchpoints don't work in device memory by Nickolai Zeldovich [1294930] change line status on floppy by Ben Lunt [1282033] SSE FXRESTORE not working correctly by Ondrej Palkovsky [816979] wget generalizations by Lyndon Nerenberg [1214886] No more pageWriteStamp / unified icache by H. Johansson [1107945] com->socket redirection support by Andrew Backer - these S.F. bugs were closed [669180] win95 install : unknown SET FEATURES subcommand 0x03 [1346692] bochs 2.2.1 VGA BIOS error [1354963] floppy in KolibriOS [1378204] error: bochs-2.2.1, --enable-sb16, --disable-gameport [1368412] VDE problems in BOCHS [533446] CPU and APIC devices appear twice [1000796] bximage fails to create image of specified size [1170793] Quarterdeck QEMM doesn't work [923704] Multiple opcode prefixes don't reflect Trap 13 [1166392] DocBook/documentation issues [1368239] broken grater than 4GB size of sparse type hd image [1365830] i386 compile breaks on paging [427550] Incomplete IRETD implementation [1215081] MSVC workspace STILL not fixed [736279] Jump to Task [1356488] FD change fail & occur error [957615] [CPU ] prefetch: RIP > CS.limit [1353866] not booting linux-2.6.14 [1351667] load32bitOSImage does not work with --enable-x86-debugger [1217476] Incorrect (?) handling of segment registers in real mode [1184711] OS2 DOS crash [2.2.pre2] [624330] support for disks > 32GiB [1348368] bochs 2.2.1 bximage error [1342081] Configuration Menu option failed [1138616] OS/2 Warp 4 hangs when booting [1049840] mouse and video conflict [1164570] Unable to perform Fedora Core 4 test 1 installation [1183201] Windows 2000 (MSDN build 2150?) does not completely install [1194284] Can't boot from CD-ROM (Windows NT) [962969] Windows NT crashes while trying to intall them. [1054594] WinXP install halts (redo) [1153107] Windows XP fails with BSOD on 'vga' [938518] Win XP installation fails [645420] getHostMemAddr vetoed direct read [1179985] MS XENIX: >>PANIC<< VGABIOS panic at vgabios.c, line 0 [1329600] WBINVD and INVD should flush caches and TLB [638924] eliminate BX_USE_CONFIG_INTERFACE [1048711] Funny behaviour with CTRL [1288450] keyboard BIOS error [1310706] Keyboard - about key SHIFT [1295981] Ubuntu 5.04 Live-CD won't boot in Bochs [879047] APIC timer behavior different before reset and after [1188506] I still can't install the german Windows XP! [1301847] Windows XP dosn't boot - FXRSTOR problem ? [661259] does not boot QNX under WinX [924412] Keyboard lock states all whacked [681127] MIPSpro compiler (IRIX) is allergic to ^M [1285923] BIOS keyboard handler [516639] ATA controller revisited... [657918] does not boot BeOS under WinX [649245] BeOS CD locks halfway on boot [1094385] Attachment for bug 1090339 (beos failure) [1183196] BeOS 4.5 developer CD does not install [1090339] BeOS fails to boot [639484] panics when int 13 is called [711701] divide by zero [704295] ATAPI/BIOS call missing [682856] hard drive problems [627691] Cursor keys problem [588011] keyboard not working [542260] os/2 warp crashes with floppy handling [1273878] SB16 doesn't work in pure DOS [542254] OS/2 FDC driver dies [1099610] Windows 98 SE Does not install [875479] cr3 problem on task switch [731423] NE2000 causing PANIC on Win2K detection [1156155] bochs fails to boot plan9 iso [1251979] --enable-cpu-level=3 should assume --without-fpu [1257538] Interupt 15h 83h - set wait event interval [658396] Panic for DR DOS emm386 [679339] /? doesn't divulge Bochs command-line syntax [1167016] call/jump/return_protected doesn't support x86-64 [1252432] Mac OS X compile bug [881442] Bochs 2.1 PANIC when loading DOS Turbo Pascal protected mode [1249324] Boch2.2.1 Buffer Overfollow in void bx_local_apic_c::init () [1197144] 'make install' has dependency on wget [1079595] LTR:386TSS: loading tr.limit < 103 [1244070] Compilation Error in gui/rfb.cc [761707] CPU error when trying to start Privateer [517281] Crash running Privateer in DOS... ------------------------------------------------------------------------- Changes in 2.2.1 (July 8, 2005): - Fixed several compilation warnings and errors for different platforms (Volker) - Fixed FPU tag word restore in FXRSTOR instruction (Stanislav) - Added missing scancodes for F11 and F12 to BIOS translation table (Volker) - Bochs disassembler bugfixes (h.johansson) - About 5% emulation speed improvement (h.johansson) - Handle writing of zero to APIC timer initial count register (Stanislav) - Enable Idle-Hack for 'TERM' GUI (h.johansson) - Reduced overhead of BX_SHOW_IPS option to minimum. Now every simulation could run with --enable-show-ips without significant performance penalty. (Stanislav) - Fixed pcipnic register access (Volker) - Limited write support for TFTP server in 'vnet' networking module added (Volker) - Changed some timing defaults to more useful values (Volker) - WinXP/2003 style common controls now supported (Vitaly Vorobyov) - Updated LGPL'd VGABIOS to version 0.5c (Volker) - Added new BX_INSTR_HLT callback to instrumentation (Stanislav) ------------------------------------------------------------------------- Changes in 2.2 (May 28, 2005): Brief summary : - New floating point emulator based on SoftFloat floating point emulation library. - improved x86-64 emulation - Cirrus SVGA card emulation added - status bar with indicators for keyboard, floppy, cdrom and disk (gui dependant) - many improvements in Bochs emulated I/O devices (e.g. PCI subsystem) Detailed change log : - CPU - fixes for booting OS/2 by Dmitri Froloff - fixed v8086 priveleged instruction processing bug (was also reported by LightCone Aug 7 2003) - exception process bug (was reported by Diego Henriquez Sat Nov 15 01:16:51 CET 2003) - segment validation with IRET instruction - CS segment not present exception processing with IRET - several fixes by Kevin Lawton - add MSVC host asm instructions (patch by suzu) - fixed bug in HADDPD/HSUBP

743

社区成员

发帖
与我相关
我的任务
社区描述
PowerBuilder 脚本语言
社区管理员
  • 脚本语言社区
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧