昔作ったPython製ライブラリのカバレッジ測ってみた

Python

かっこ悪くて面倒でもテストコードを書こうを読んで。

私は職業プログラマではありませんが、unittestでテストを書きながらコーディングすれば良い物が出来るという体感はありました。で、カバレッジって何だ？どうやって測るんだ？って思って調べてみると、Pythonではcoverageパッケージを使えばC0: 命令網羅率(ステートメントカバレッジ)が簡単に測れるとのこと。

それでは昔作ったPython製ライブラリ（PyAmazonCloudDrive）でやってみよー！

coverageのインストール

easy_installするだけ

C:\temp>easy_install coverage
Searching for coverage
Reading http://pypi.python.org/simple/coverage/
Reading http://nedbatchelder.com/code/modules/coverage.html
Reading http://nedbatchelder.com/code/coverage
Reading http://nedbatchelder.com/code/coverage/3.5b1
Reading http://nedbatchelder.com/code/coverage/3.4b1
Reading http://nedbatchelder.com/code/coverage/3.4b2
Reading http://nedbatchelder.com/code/coverage/3.5.1b1
Best match: coverage 3.5.1
Downloading http://pypi.python.org/packages/2.7/c/coverage/coverage-3.5.1.win32-py2.7.exe#md5=04b9eb49d9e34a5b7310550f657a8a27
Processing coverage-3.5.1.win32-py2.7.exe
Deleting c:\users\sakurai\appdata\local\temp\easy_install-wo97iz\coverage-3.5.1-py2.7-win32.egg.tmp\EGG-INFO\scripts\coverage-script.py
Deleting c:\users\sakurai\appdata\local\temp\easy_install-wo97iz\coverage-3.5.1-py2.7-win32.egg.tmp\EGG-INFO\scripts\coverage.exe
Deleting c:\users\sakurai\appdata\local\temp\easy_install-wo97iz\coverage-3.5.1-py2.7-win32.egg.tmp\EGG-INFO\scripts\coverage.exe.manifest
creating 'c:\users\sakurai\appdata\local\temp\easy_install-wo97iz\coverage-3.5.1-py2.7-win32.egg' and adding 'c:\users\sakurai\appdata\local\temp\easy_install-wo97iz\coverage-3.5.1-py2.7-win32.egg.tmp' to it
creating c:\python27\lib\site-packages\coverage-3.5.1-py2.7-win32.egg
Extracting coverage-3.5.1-py2.7-win32.egg to c:\python27\lib\site-packages
Adding coverage 3.5.1 to easy-install.pth file
Installing coverage-script.pyc script to C:\Python27\Scripts
Installing coverage-script.py script to C:\Python27\Scripts
Installing coverage.exe script to C:\Python27\Scripts
Installing coverage.exe.manifest script to C:\Python27\Scripts

Installed c:\python27\lib\site-packages\coverage-3.5.1-py2.7-win32.egg
Processing dependencies for coverage
Finished processing dependencies for coverage

使い方

eraseで統計データの初期化、runでテストコードの実行/計測、reportでカバレッジの計測結果の表示、htmlでHTMLレポート作成とのこと。（※自身の環境ではC:\Python27\Scriptsにパスを通してます。）

C:\temp>coverage --help
Coverage.py, version 3.5.1
Measure, collect, and report on code coverage in Python programs.

usage: coverage <command> [options] [args]

Commands:
    annotate    Annotate source files with execution information.
    combine     Combine a number of data files.
    erase       Erase previously collected coverage data.
    help        Get help on using coverage.py.
    html        Create an HTML report.
    report      Report coverage stats on modules.
    run         Run a Python program and measure code execution.
    xml         Create an XML report of coverage results.

Use "coverage help <command>" for detailed help on any command.
Use "coverage help classic" for help on older command syntax.
For more information, see http://nedbatchelder.com/code/coverage

やってみる

予想外にテストでERRORが発生しましたが、まぁ良しとします。例外が発生したおかげもあって（？）全体として85%のカバレッジらしいです。個人的には、まぁまぁな数字じゃないかと思います。

  1. coverage erase（統計データの初期化：一応）
  2. coverage run test.py（テストコードの実行/計測：コマンドライン引数も問題なし）
  3. coverage report（カバレッジの計測結果の表示：引数なしで全スクリプトの計測結果を表示）
  4. coverage html（HTMLレポート作成：htmlcovというフォルダに生成されます）
C:\temp\pyacd>coverage erase

C:\temp\pyacd>coverage run test.py someone@example.com password
*** Inputs are here ***
email: someone@example.com
password: password
********************

testLogin (__main__.AuthTest) ... GET->GET->GET->GET->POST->GET->ok

----------------------------------------------------------------------
Ran 1 test in 9.873s

OK
testEmptyRecycleBin (__main__.ApiTest) ... GET->ok
testFile_Create_Upload_Download (__main__.ApiTest) ... GET->GET->POST->GET->GET->GET->GET->ok
testFolder_Create_Rename_Copy_Recycle_Remove (__main__.ApiTest) ... GET->GET->GET->GET->GET->GET->GET->GET->ERROR
testInfoByPathAndById (__main__.ApiTest) ... GET->GET->ok
testListById (__main__.ApiTest) ... GET->GET->ok
testSubscriptionProblem (__main__.ApiTest) ... GET->ok
testUserStorage (__main__.ApiTest) ... GET->ok

======================================================================
ERROR: testFolder_Create_Rename_Copy_Recycle_Remove (__main__.ApiTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 144, in testFolder_Create_Rename_Copy_Recycle_Remove
    pyacd.api.remove_bulk_by_id([folder1.object_id,])
  File "C:\temp\pyacd\pyacd\api.py", line 259, in remove_bulk_by_id
    _operate2_bulk_by_id("removeBulkById",source_inclusion_ids)
  File "C:\temp\pyacd\pyacd\api.py", line 278, in _operate2_bulk_by_id
    _error_check(resp_json)
  File "C:\temp\pyacd\pyacd\api.py", line 82, in _error_check
    raise pyacd.PyAmazonCloudDriveApiException(resp_json.get("Error"))
PyAmazonCloudDriveApiException: PyAmazonCloudDriveError: MetadataNotFoundException:87940d81-1e17-41c8-a99e-9c660f213aa9 does not exist.

----------------------------------------------------------------------
Ran 7 tests in 19.612s

FAILED (errors=1)

C:\temp\pyacd>coverage report
Name                Stmts   Miss  Cover
---------------------------------------
pyacd\__init__         19      0   100%
pyacd\api             197     31    84%
pyacd\apiresponse      95     16    83%
pyacd\auth             87     10    89%
pyacd\connection       84     35    58%
pyacd\exception        16      2    88%
pyacd\multipart        46      2    96%
pyacd\status            3      0   100%
pyacd\types             5      0   100%
test                   81      2    98%
---------------------------------------
TOTAL                 633     98    85%

C:\temp\pyacd>coverage html

HTMLレポートを見てみる

テストできていない部分（未実行のコード）が赤でハイライトされるため一目瞭然です。個人的には各クラスの__repr__や__str__のテストが抜ける傾向にあるんだなぁと気付きが有りました。


まとめ

coverageパッケージはかなりイイです。なんとなく書いたテストがどの程度の網羅性を持っているか、定量的に図れるメリットは非常に大きいと思います。「coverage run」でどんなスクリプトでも計測できるので、これからは必ずこのパッケージを使っていこうと思いました（マル）

追記：分岐網羅(C1)について

id:imagawa_yakataさんにコメントで教えて頂きましたC1:分岐網羅について。「coverage run」に「--branch」オプションをつけると、分岐の網羅性についても計測出来るとのこと。ということで早速、C1分岐網羅性の測定もやったところ82%でした。これもまずまず（？）かなぁと。。。

C:\temp\pyacd>coverage run --branch test.py
*** Inputs are here ***
email: 
password: 
********************

testLogin (__main__.AuthTest) ... GET->GET->GET->GET->POST->GET->ok

----------------------------------------------------------------------
Ran 1 test in 9.134s

OK
testEmptyRecycleBin (__main__.ApiTest) ... GET->ok
testFile_Create_Upload_Download (__main__.ApiTest) ... GET->GET->POST->GET->GET->GET->GET->ok
testFolder_Create_Rename_Copy_Recycle_Remove (__main__.ApiTest) ... GET->GET->GET->GET->GET->GET->GET->GET->ERROR
testInfoByPathAndById (__main__.ApiTest) ... GET->GET->ok
testListById (__main__.ApiTest) ... GET->GET->ok
testSubscriptionProblem (__main__.ApiTest) ... GET->ok
testUserStorage (__main__.ApiTest) ... GET->ok

======================================================================
ERROR: testFolder_Create_Rename_Copy_Recycle_Remove (__main__.ApiTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 144, in testFolder_Create_Rename_Copy_Recycle_Remove
    pyacd.api.remove_bulk_by_id([folder1.object_id,])
  File "C:\temp\pyacd\pyacd\api.py", line 259, in remove_bulk_by_id
    _operate2_bulk_by_id("removeBulkById",source_inclusion_ids)
  File "C:\temp\pyacd\pyacd\api.py", line 278, in _operate2_bulk_by_id
    _error_check(resp_json)
  File "C:\temp\pyacd\pyacd\api.py", line 82, in _error_check
    raise pyacd.PyAmazonCloudDriveApiException(resp_json.get("Error"))
PyAmazonCloudDriveApiException: PyAmazonCloudDriveError: MetadataNotFoundException:f5a962d0-5578-4ce1-8c3c-c2a85bd8017c does not exist.

----------------------------------------------------------------------
Ran 7 tests in 21.944s

FAILED (errors=1)

C:\temp\pyacd>coverage report
Name                Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------
pyacd\__init__         19      0      0      0   100%
pyacd\api             197     31     40     18    79%
pyacd\apiresponse      95     16      8      1    83%
pyacd\auth             87     10     25      4    88%
pyacd\connection       84     35     36      9    63%
pyacd\exception        16      2      2      1    83%
pyacd\multipart        46      2     10      3    91%
pyacd\status            3      0      0      0   100%
pyacd\types             5      0      0      0   100%
test                   81      2      6      2    95%
-----------------------------------------------------
TOTAL                 633     98    127     38    82%