wal: bound record size and tighten repair copy permissions

[pfi79]

A corrupt or forged record header could claim 0xFFFFFFFF bytes and drive a ~4GB allocation on Open. Reject such records with ErrRecordTooBig, both on read (against FileSizeBytes) and on append. Repair copies are now 0600, and an existing .copy gets chmod'ed, since WriteFile does not touch its mode.

[Copilot]

🟡 Changes recommended

Repair can truncate valid large records, and pre-existing copy files briefly expose WAL data before permissions are tightened.

2 open findings

• Preserve configured WAL size limit during repair · New
• Set restrictive file mode before copying WAL data · New

What changed in this PR

Bounds WAL record sizes to prevent oversized allocations and secures repair-copy permissions.

Changes:

• Adds ErrRecordTooBig and enforces record-size limits during reads and appends.
• Makes repair copies owner-only and adds regression tests.

File	Description
pkg/​wal/​writeaheadlog.go	Enforces configured record limits.
pkg/​wal/​reader.go	Adds bounded record reading.
pkg/​wal/​reader_test.go	Tests oversized records.
pkg/​wal/​util.go	Tightens repair-copy permissions.
pkg/​wal/​util_test.go	Tests private repair copies.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[HagarMeir]

no reason for this change here in this PR

[pfi79]

fix

[C0rWin]

ErrRecordTooBig on the last file should go through repair like the other torn-write errors — ReadAll (lines 576-578) doesn't list it, so what used to surface as io.ErrUnexpectedEOF (short ReadFull) and auto-repair now fails startup. scanRepairFile already truncates on any non-EOF error, it just needs to be reachable.

		if index == lastIndex &&
			(errors.Is(readErr, io.ErrUnexpectedEOF) ||
				errors.Is(readErr, ErrCRC) ||
				errors.Is(readErr, ErrWALUnmarshalPayload) ||
				errors.Is(readErr, ErrRecordTooBig)) {

[pfi79]

fix

[C0rWin]

Worth a case in TestWriteAheadLogFile_Repair as well — assertTestRepair only does assert.Error on the first ReadAll, so it can't tell a repairable error from a fatal one. This fails on the current branch and passes with the ReadAll change:

t.Run("oversized record header", func(t *testing.T) {
	dirPath := filepath.Join(testDir, "oversized")
	assert.NoError(t, os.MkdirAll(dirPath, walDirPermPrivateRWX))

	wal, err := Create(logger, dirPath, &Options{FileSizeBytes: 10 * 1024, BufferSizeBytes: 2048})
	assert.NoError(t, err)

	const NumBytes = 1024
	const NumRec = 102
	data1 := make([]byte, NumBytes)
	for m := range NumRec {
		for n := range NumBytes {
			data1[n] = byte(m)
		}
		assert.NoError(t, wal.Append(data1, false))
	}
	assert.NoError(t, wal.Close())

	// append a header claiming a ~4GB record to the last file
	names, err := dirReadWalNames(dirPath)
	assert.NoError(t, err)
	lastFile := filepath.Join(dirPath, names[len(names)-1])
	f, err := os.OpenFile(lastFile, os.O_RDWR, walFilePermPrivateRW)
	assert.NoError(t, err)
	_, err = f.Seek(0, io.SeekEnd)
	assert.NoError(t, err)
	header := make([]byte, recordHeaderSize)
	binary.LittleEndian.PutUint64(header, 0xFFFFFFFF)
	_, err = f.Write(header)
	assert.NoError(t, err)
	assert.NoError(t, f.Close())

	wal, err = Open(logger, dirPath, &Options{FileSizeBytes: 10 * 1024, BufferSizeBytes: 2048})
	assert.NoError(t, err)
	_, err = wal.ReadAll()
	assert.ErrorIs(t, err, io.ErrUnexpectedEOF)
	assert.NoError(t, wal.Close())

	assertTestRepair(t, logger, dirPath, NumRec)
})

[pfi79]

fix

[tock-ibm]

Good comment

[tock-ibm]

Note that trying to repair on any of these errors may lead the application (fabric) to start smartbft in a state from the past even before a committed record (block). For example, if the file is corrupt somewhere before a TruncateTo(true) control record, you may succeed to "repair" the file, but the end result will be you start smartbft with a wal record that was already committed.

The repair mechanism is against a torn write on the last record, repairing any other corruption is not so easy. We assume that the error is on the last record only. Any other corruption is not necessarily repairable.

Take a look at the test cases in TestWriteAheadLogFile_Repair - only on the tail.

The safe option is to throw away the node and start a new one.

[pfi79]

I suggest checking and fixing it only if the error is in the tail. Throw out the rest.

[pfi79]

fix