Working with JavaExtractor directly

[CaledoniaProject]

I'm playing with JavaExtractor and it fails at the import step, does anyone know what's wrong?

export CODEQL_DIST=/usr/local/homebrew/Caskroom/codeql/2.9.0/codeql

export CODEQL_JAVA_HOME=$CODEQL_DIST/tools/osx64/java/
export CODEQL_EXTRACTOR_JAVA_LOG_DIR=testdb/log
export CODEQL_EXTRACTOR_JAVA_ROOT=$CODEQL_DIST/java
export CODEQL_EXTRACTOR_JAVA_SOURCE_ARCHIVE_DIR=test-db/src
export CODEQL_EXTRACTOR_JAVA_TRAP_DIR=test-db/trap/java
export CODEQL_EXTRACTOR_JAVA_WIP_DATABASE=$PWD/test-db

codeql database init test-db -l java --source-root src --overwrite

find src -name '*.java' > javac-args
java -cp $CODEQL_DIST/java/tools/semmle-extractor-java.jar com.semmle.extractor.java.JavaExtractor --javac-args javac-args

codeql database import test-db/db-java test-db/trap -S $CODEQL_DIST/java/semmlecode.dbscheme
codeql database finalize test-db

[MathiasVP]

Hi @CaledoniaProject,

Do you get any kind of error message in the import step?

[CaledoniaProject]

I got A fatal error occurred: XXX/test-db/trap is not a recognized CodeQL database. In the test-db folder only one file exists: test-db/trap/java/diagnostics/diagnostic.trap.gz

[aibaars]

I think there is no need to run the codeql database import step. The codeql database finalize test-db should take care of importing the TRAP files.

[MathiasVP]

I'm not quite sure why you're doing these init and finalize steps manually instead of just doing codeql database create, but depending on your use-case you might find codeql database trace-command useful. You can check out codeql database trace-command --help for information on this subcommand.

[CaledoniaProject]

I'm trying to make a database from close-source jars. So I first use java decompiler to generate the *.java files then manually use the steps to create a codeql database.

[CaledoniaProject]

Executed again on a Linux server, here's the error I got:

First decompile a target jar, e.g jline-0.9.94.jar

# mkdir src
# /usr/bin/java -jar fernflower.jar -hdc=0 -dgs=1 -rsy=1 -rbr=1 -lit=1 -nls=1 -mpm=60 jline-0.9.94.jar src
...
INFO:  Decompiling class jline/UnixTerminal
INFO:  ... done
INFO:  Decompiling class jline/UnsupportedTerminal
INFO:  ... done
INFO:  Decompiling class jline/WindowsTerminal
INFO:  ... done

Then I extract the source zip in the "src" folder:

# (cd src; unzip *.jar; ls)
./  ../  jline/  jline-0.9.94.jar  META-INF/

Now I create the codeql database:

export CODEQL_DIST=/home/opt/codeql

export CODEQL_JAVA_HOME=$CODEQL_DIST/tools/linux64/java/
export CODEQL_EXTRACTOR_JAVA_LOG_DIR=test-db/log
export CODEQL_EXTRACTOR_JAVA_ROOT=$CODEQL_DIST/java
export CODEQL_EXTRACTOR_JAVA_SOURCE_ARCHIVE_DIR=test-db/src
export CODEQL_EXTRACTOR_JAVA_TRAP_DIR=test-db/trap/java
export CODEQL_EXTRACTOR_JAVA_WIP_DATABASE=$PWD/test-db

# codeql database init test-db -l java --source-root src --overwrite
Counting lines of code in /home/tmp/codeql-test/src
Resolving extractor java.
Successfully loaded extractor Java (java) from /home/opt/codeql/java.
Created skeleton CodeQL database at /home/tmp/codeql-test/test-db. This in-progress database is ready to be populated by an extractor.

# find src -name '*.java' > javac-args
# java -cp $CODEQL_DIST/java/tools/semmle-extractor-java.jar com.semmle.extractor.java.JavaExtractor --javac-args javac-args
# head javac-args
src/jline/CursorBuffer.java
src/jline/UnixTerminal.java
src/jline/CandidateListCompletionHandler.java
src/jline/CompletionHandler.java
src/jline/Terminal.java
src/jline/ConsoleRunner.java
src/jline/SimpleCompletor.java
src/jline/MultiCompletor.java
src/jline/WindowsTerminal.java
src/jline/CandidateCycleCompletionHandler.java

# codeql database import test-db/db-java test-db/trap -S $CODEQL_DIST/java/semmlecode.dbscheme
A fatal error occurred: Database root /home/tmp/codeql-test/test-db/db-java does not exist.

# codeql database finalize test-db
No source code was seen and extracted to /home/tmp/codeql-test/test-db.
This can occur if the specified build commands failed to compile or process any code.
 - Confirm that there is some source code for the specified language in the project.
 - For codebases written in Go, JavaScript, TypeScript, and Python, do not specify
   an explicit --command.
 - For other languages, the --command must specify a "clean" build which compiles
   all the source code files without reusing existing build artefacts.

[aibaars]

Have you checked test-db/log ? The problem is likely that you need to prefix the java-args argument with a @ (--javac-args @javac-args).

I think you could simply use codeql database create test-db -s src -ljava --command "javac @java-args". That way there is no need to use all the internal commands and environment variables.