CryptoAPI-bench: A comprehensive benchmark on java cryptographic API misuses

Sharmin Afrose, Sazzadur Rahaman, Danfeng Daphne Yao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

23 Scopus citations


Several studies showed that misuses of cryptographic APIs are common in real-world code (e.g., Apache projects and Android apps). There exist several open-sourced and commercial security tools that automatically screen Java programs to detect misuses. In order to compare their accuracy and security guarantees, we develop a comprehensive benchmark named CryptoAPI-Bench. CryptoAPI-Bench consists of 171 unit test cases that cover basic cases, as well as complex cases, including interprocedural, field sensitive, multiple class test cases, and path sensitive data flow of misuse cases. The benchmark also includes correct cases for testing false positive rates. We evaluate CryptoAPI-Bench on four tools, namely, SpotBugs, CryptoGuard, CrySL, and Coverity and present their performance and comparative analysis. Our benchmark is useful for advancing state-of-the-art solutions in the space of misuse detection.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE Secure Development, SecDev 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages13
ISBN (Electronic)9781538672891
StatePublished - Sep 2019
Externally publishedYes
Event2019 IEEE Secure Development, SecDev 2019 - McLean, United States
Duration: Sep 25 2019Sep 27 2019

Publication series

NameProceedings - 2019 IEEE Secure Development, SecDev 2019


Conference2019 IEEE Secure Development, SecDev 2019
Country/TerritoryUnited States


  • Accuracy
  • Benchmark
  • Cryptographic API misuses

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software
  • Safety, Risk, Reliability and Quality


Dive into the research topics of 'CryptoAPI-bench: A comprehensive benchmark on java cryptographic API misuses'. Together they form a unique fingerprint.

Cite this