mmap leak and invalid JNI access after `NewDirectByteBuffer()` failure

[K-ANOY]

I found a possible JNI error-handling issue in the shared-memory mapping helpers. After a native mapping has been created, NewDirectByteBuffer() is called to expose it to Java. If that call fails with a pending exception, the code continues and uses the null ByteBuffer object instead of returning and cleaning up the mapping.

File: junixsocket-native/src/main/c/shm.c

Functions: Java_org_newsclub_net_unix_NativeUnixSocket_mmapShm, Java_org_newsclub_net_unix_NativeUnixSocket_mappedBuffer

Relevant code from mmapShm():

addr = mmap(NULL, (size_t)length, prot, flags, handle, (off_t)offset);
if(addr == MAP_FAILED) {
    throwIOErrnumException(env, errno, NULL);
    return NULL;
} else if(addr == NULL) {
    _throwException(env, kExceptionIOException, "mmap");
    return NULL;
}

// NewDirectByteBuffer is guaranteed to create a DirectByteBuffer class, which is a subclass of MappedByteBuffer/Buffer
jobject dbb = (*env)->NewDirectByteBuffer(env, addr, length);
if(dbb == NULL) {
    if(!((*env)->ExceptionCheck(env))) {
        _throwException(env, kExceptionIOException, "NewDirectByteBuffer");
        return NULL;
    }
}

// enable mmap-specific operations on MemorySegment (isMapped=true -> force(), load(), isLoaded(), unload())
// MappedByteBuffer.fd
if(kMappedByteBufferFieldFd && haveFd) {
    (*env)->SetObjectField(env, dbb, kMappedByteBufferFieldFd, fd);
}

NewDirectByteBuffer() may return NULL, for example if the VM cannot allocate the Java direct-buffer object. When that happens with a pending exception, the current check does not return:

if(dbb == NULL) {
    if(!((*env)->ExceptionCheck(env))) {
        _throwException(env, kExceptionIOException, "NewDirectByteBuffer");
        return NULL;
    }
}

So execution continues and later JNI calls use dbb even though it is NULL:

(*env)->SetObjectField(env, dbb, kMappedByteBufferFieldFd, fd);
(*env)->SetBooleanField(env, dbb, kMappedByteBufferFieldIsSync, JNI_TRUE);
(*env)->SetObjectField(env, dbb, kBufferFieldSegment, arenaSegment);

That is invalid JNI usage: the object argument to these field writes must be a valid object, not NULL.

There is also a native resource leak in the mmapShm() path. The mapping has already been created before NewDirectByteBuffer() is called. If the direct-buffer object cannot be created, Java never receives a ByteBuffer and therefore cannot register or run the normal cleanup path. The native code also does not call munmap() before returning or propagating the pending exception, so the successful mapping can be leaked.

The same missing early return pattern exists in mappedBuffer():

jobject dbb = (*env)->NewDirectByteBuffer(env, (void*)addr, length);
if(dbb == NULL) {
    if(!((*env)->ExceptionCheck(env))) {
        _throwException(env, kExceptionIOException, "NewDirectByteBuffer");
        return NULL;
    }
}

if(kMappedByteBufferFieldFd && fd != NULL) {
    (*env)->SetObjectField(env, dbb, kMappedByteBufferFieldFd, fd);
}

Suggested fix: return immediately whenever NewDirectByteBuffer() returns NULL. In mmapShm(), also release the native mapping before returning on this failure path, for example by calling the same unmap logic used by NativeUnixSocket_unmap().