Add support for parsing Git commit messages

[ziadhany]

I created an initial script to parse Git commit messages that can be easily integrated with our model. The script takes a Git repository as input, parses all commits, and returns the CVEs along with their corresponding fixed commits.

Issues:

• Review and add CVE fixes dataset #1383
• fedcode-next: Extract fix commits from the commit logs in search for CVE-related commit messages #2000
• Extract unpublished vulnerabilities from commit histories and trackers #1129

results:
vulnerabilities_packagecommitpatch2.zip

[ziadhany]

This part should not be static

[keshav-space]

Thanks @ziadhany, see some suggestions.

[ziadhany]

This is the list of repositories I think we should collect and parse Git messages from:

• Linux Kernel: https://github.com/torvalds/linux
• BusyBox: https://github.com/mirror/busybox.git
• Nginx: https://github.com/nginx/nginx.git
• Apache Tomcat: https://github.com/apache/tomcat.git
• MySQL Server: https://github.com/mysql/mysql-server.git
• PostgreSQL: https://github.com/postgres/postgres.git
• MongoDB: https://github.com/mongodb/mongo.git
• Redis: https://github.com/redis/redis.git
• SQLite: https://github.com/sqlite/sqlite.git
• PHP: https://github.com/php/php-src.git
• Python (CPython): https://github.com/python/cpython.git
• Ruby: https://github.com/ruby/ruby.git
• Go (Golang): https://github.com/golang/go.git
• Node.js: https://github.com/nodejs/node.git
• Rust: https://github.com/rust-lang/rust.git
• OpenJDK (Java): https://github.com/openjdk/jdk.git
• Swift: https://github.com/swiftlang/swift.git
• Django: https://github.com/django/django.git
• Rails: https://github.com/rails/rails.git
• Laravel: https://github.com/laravel/framework.git
• Spring Framework: https://github.com/spring-projects/spring-framework.git
• React: https://github.com/facebook/react
• Angular: https://github.com/angular/angular
• WordPress: https://github.com/WordPress/WordPress.git
• Docker (Moby): https://github.com/moby/moby.git
• Kubernetes: https://github.com/kubernetes/kubernetes.git
• QEMU: https://gitlab.com/qemu-project/qemu.git
• Xen Project: https://github.com/xen-project/xen.git
• VirtualBox: https://github.com/mirror/vbox.git
• Containerd: https://github.com/containerd/containerd.git
• Ansible: https://github.com/ansible/ansible.git
• Terraform: https://github.com/hashicorp/terraform.git
• Wireshark: https://gitlab.com/wireshark/wireshark.git
• Tcpdump: https://github.com/the-tcpdump-group/tcpdump.git
• Git: https://github.com/git/git.git
• Jenkins: https://github.com/jenkinsci/jenkins.git
• GitLab (Community Edition): https://gitlab.com/gitlab-org/gitlab-foss.git

Ignored:

• OpenSSL: https://github.com/openssl/openssl.git

[keshav-space]

@ziadhany thanks, see some feedback below.

[keshav-space]

Thanks @ziadhany, LGTM! Do we have an estimate of how long it takes to run CollectLinuxFixCommitsPipeline?

[ziadhany]

Thanks @ziadhany, LGTM! Do we have an estimate of how long it takes to run CollectLinuxFixCommitsPipeline?

The Linux kernel pipeline takes around 5 minutes if the network call is ignored.