Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ tinyShieldスクリプトは、Adblockコミュニティによって管理され
### クイックスタート
このユーザースクリプトを検出するには、次のURLをクリックしてください。

生成されたスクリプトは標準の `@match` を使用します。元データの `.*` パターンは、具体的なドメインも収集されない限り除外されます。

https://cdn.jsdelivr.net/npm/@filteringdev/tinyshield@latest/dist/tinyShield.user.js

### ドメイングループ購読ガイド (Apple Safari Extension ユーザー専用)
Expand Down
4 changes: 4 additions & 0 deletions README.ko.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ tinyShield 유저스크립트는 애드블록 커뮤니티에 의해 관리되
### 빠른 시작
아래 URL를 클릭하여 설치해주세요:

생성된 스크립트는 표준 `@match`를 사용합니다. 원본의 `.*` 패턴은 구체적인 도메인이 함께 수집되지 않으면 제외됩니다.

https://cdn.jsdelivr.net/npm/@filteringdev/tinyshield@latest/dist/tinyShield.user.js

### 도메인 그룹 구독 가이드 (Apple Safari Extension 사용자 전용)
Expand All @@ -46,6 +48,8 @@ https://cdn.jsdelivr.net/npm/@filteringdev/tinyshield@latest/dist/tinyShield.use

이 도메인 그룹 분리는 https://github.com/FilteringDev/tinyShield/issues/38 의 영향 범위를 가능한 한 최소화하기 위한 목적입니다.

`metadata.json`은 수집된 구체 도메인을 그룹 파일명에 연결합니다. `example.tld` 같은 그룹 이름이 모든 접미사의 실행을 허용하는 것은 아닙니다. 도메인 그룹이 바뀌면 그룹 파일명도 바뀔 수 있습니다. `SHA386`은 대문자 SHA-384 해시입니다.

### 설치하는 법
Firefox 및 Chromium 기반 브라우저에서는 **Violentmonkey**를, macOS·iOS·iPadOS의 Safari에서는 **wBlock**을 권장합니다. **Tampermonkey는 권장하지 않습니다.**

Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ If you believe this userscript is causing issues while in use, please use GitHub
### Quick Start
Just click the following URL to detect this userscript.

The generated scripts use standard `@match` rules. Source patterns ending in `.*` are excluded unless concrete domains are also collected.

https://cdn.jsdelivr.net/npm/@filteringdev/tinyshield@latest/dist/tinyShield.user.js

### Domain Group Subscription Guide (Apple Safari Extension Only)
Expand All @@ -46,6 +48,8 @@ https://cdn.jsdelivr.net/npm/@filteringdev/tinyshield@latest/dist/tinyShield.use

This domain-group split is intended to reduce the impact scope of https://github.com/FilteringDev/tinyShield/issues/38 as much as possible.

`metadata.json` maps collected concrete domains to group filenames. Generated group names such as `example.tld` do not themselves authorize every suffix. Group filenames may change when the domain groups change. `SHA386` contains the uppercase SHA-384 digest.

### How to install
**Violentmonkey** is recommended for Firefox and Chromium-based browsers. **wBlock** is recommended for Safari on macOS, iOS, and iPadOS. **Tampermonkey is not recommended.**

Expand Down
5 changes: 4 additions & 1 deletion builder/source/banner/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ export interface BannerOptions {
}

export function CreateBanner(Options: BannerOptions): string {
const Domains = [...Options.Domains].filter(Domain => !Domain.endsWith('.*'))
if (Domains.length === 0) throw new Error('No concrete domains available for userscript matching.')

let BannerString: string = '// ==UserScript==\n'
BannerString += `// @name ${Options.BuildType === 'production' ? Options.Name : Options.Name + ' (Development)'}\n`
BannerString += '//\n'
Expand All @@ -38,7 +41,7 @@ export function CreateBanner(Options: BannerOptions): string {
}
BannerString += '//\n'

for (const Domain of Options.Domains) {
for (const Domain of Domains) {
BannerString += `// @match *://${Domain}/*\n`
BannerString += `// @match *://*.${Domain}/*\n`
}
Expand Down
45 changes: 23 additions & 22 deletions builder/source/build-grouped.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,17 @@ export class GroupedBuild extends Build {
super(FromOrOption)
}

private TransformDomainSetToFileName(DomainSet: Set<Set<string>>): Set<Set<string>> {
const Result = new Set<Set<string>>()
for (const Value of DomainSet) {
const FileNameSet = new Set<string>()
for (const Domain of Value) {
FileNameSet.add(Domain.endsWith('.*') ? Domain.replaceAll(/\.\*$/g, '.tld') : Domain)
}
Result.add(FileNameSet)
private GetGroups(): Map<Set<string>, Set<string>> {
const Groups = this.FetchedDomains.get('Groups')
if (!Groups) throw new Error('Domain groups are not initialized.')
const Result = new Map<Set<string>, Set<string>>()
for (const [Names, Domains] of Groups) {
const MatchingDomains = new Set([...Domains].filter(Domain => !Domain.endsWith('.*')))
if (MatchingDomains.size === 0) continue
Result.set(
new Set([...Names].map(Domain => Domain.endsWith('.*') ? Domain.slice(0, -2) + '.tld' : Domain)),
MatchingDomains
)
}
return Result
}
Expand All @@ -38,12 +41,8 @@ export class GroupedBuild extends Build {
FileMapping: {}
}

const FileNameList: Set<Set<string>> = this.TransformDomainSetToFileName(new Set<Set<string>>(this.FetchedDomains.get('EachDomain')))
const MatchingDomains: Set<Set<string>> = new Set<Set<string>>(this.FetchedDomains.get('EachDomainFull'))

for (let I = 0; I < FileNameList.size; I++) {
const FileName = [...[...FileNameList][I]][0]
const Domains = [...MatchingDomains][I]
for (const [Names, Domains] of this.GetGroups()) {
const FileName = Names.values().next().value!
const FilePath = `${this.ProjectRoot}/dist/grouped/${FileName[0]}/tinyShield-${FileName}.user.js`
const FileContent = await Fs.promises.readFile(FilePath)
const SHA386Hash = Crypto.createHash('sha384').update(FileContent).digest('hex').toUpperCase()
Expand All @@ -57,6 +56,7 @@ export class GroupedBuild extends Build {
}
}

await Fs.promises.mkdir(`${this.ProjectRoot}/dist/grouped`, { recursive: true })
await Fs.promises.writeFile(`${this.ProjectRoot}/dist/grouped/metadata.json`, JSON.stringify(MetadataObj, null, 2), 'utf-8')
}

Expand All @@ -67,19 +67,20 @@ export class GroupedBuild extends Build {

this.Options.Version ??= (await PackageJson.load(this.ProjectRoot)).content.version ?? '0.0.0'

const FileNameList: Set<Set<string>> = this.TransformDomainSetToFileName(new Set<Set<string>>(this.FetchedDomains.get('EachDomain')))
const MatchingDomains: Set<Set<string>> = new Set<Set<string>>(this.FetchedDomains.get('EachDomainFull'))

const Workerpool = new Piscina.Piscina({
filename: Path.resolve(this.ProjectRoot, 'builder/source/build-grouped-worker.ts'),
execArgv: [...Process.execArgv, '--import=tsx'],
execArgv: [...Process.execArgv, `--import=${import.meta.resolve('tsx')}`],
workerData: { Config: { ...this.Options, SubscriptionUrl: this.Options.SubscriptionUrl.href }, ProjectRoot: this.ProjectRoot }
})

const WorkerResults: Promise<void>[] = []
for (let I = 0; I < FileNameList.size; I++) {
WorkerResults.push(Workerpool.run({ FileName: [...FileNameList][I], Domains: [...MatchingDomains][I] }))
try {
const WorkerResults: Promise<unknown>[] = []
for (const [FileName, Domains] of this.GetGroups()) {
WorkerResults.push(Workerpool.run({ FileName, Domains }))
}
await Promise.all(WorkerResults)
} finally {
await Workerpool.destroy()
}
await Promise.all(WorkerResults)
}
}
54 changes: 23 additions & 31 deletions libs/source/references/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ import { CustomDefinedMatches, CustomExcludeMatches } from './custom-defined.js'
import { FetchAdShieldDomainsFromFiltersLists } from './filterslists.js'
import { FetchIABSellersJsonData } from './iabsellers.js'
import { DiscardResolvedDupWildcard } from './utils/discard-resolved-dup-wildcard.js'
import { RegroupDomainTldLevel } from './utils/regroup-domain-tldlevel.js'
import { ConvertWildcardSuffixToRegexPattern } from './utils/wildcard-suffix-converter.js'
import { GetDomainRootLabel, RegroupDomainTldLevel } from './utils/regroup-domain-tldlevel.js'

export { CustomDefinedMatches, CustomExcludeMatches } from './custom-defined.js'
export { FetchAdShieldDomainsFromFiltersLists } from './filterslists.js'
Expand All @@ -14,31 +13,35 @@ export { IndexAdShieldDomainsFromUBO } from './filterslists/uBO.js'
export { AdShieldCDNDomains, IsAdShieldCDNDomain } from './filterslists/keywords.js'
export { DiscardResolvedDupWildcard } from './utils/discard-resolved-dup-wildcard.js'
export { RegroupDomainTldLevel } from './utils/regroup-domain-tldlevel.js'
export { ConvertWildcardSuffixToRegexPattern, PublicSuffixList } from './utils/wildcard-suffix-converter.js'

export type TASDomainContainer =
Map<'Normal', Set<string>> &
Map<'Full', Set<string>> &
Map<'EachDomain', Set<Set<string>>> &
Map<'EachDomainFull', Set<Set<string>>>
Map<'EachDomainFull', Set<Set<string>>> &
Map<'Groups', Map<Set<string>, Set<string>>>

function ConvertToFlatFullDomains(Origin: Set<string>): Set<string> {
return new Set<string>([...Origin].flatMap(Domain => ConvertWildcardSuffixToRegexPattern(Domain)))
}

function IsGroupedDomains(Origin: Set<string> | Set<Set<string>>): Origin is Set<Set<string>> {
const FirstValue = Origin.values().next().value
return typeof FirstValue !== 'string' && typeof FirstValue !== 'undefined'
}

function ConvertToFullDomains(Origin: Set<string>): Set<string>
function ConvertToFullDomains(Origin: Set<Set<string>>): Set<Set<string>>
function ConvertToFullDomains(Origin: Set<string> | Set<Set<string>>): Set<string> | Set<Set<string>> {
if (IsGroupedDomains(Origin)) {
return new Set<Set<string>>([...Origin].map(DomainGroup => ConvertToFlatFullDomains(DomainGroup)))
export function CreateDomainViews(Domains: Set<string>): TASDomainContainer {
const FullDomains = new Set([...Domains].sort())
const NormalDomains = DiscardResolvedDupWildcard(FullDomains)
const EachDomain = RegroupDomainTldLevel(NormalDomains)
const MatchingGroups = new Map<string, Set<string>>()
for (const Domain of FullDomains) {
const Root = GetDomainRootLabel(Domain)
const Group = MatchingGroups.get(Root) ?? new Set<string>()
Group.add(Domain)
MatchingGroups.set(Root, Group)
}

return ConvertToFlatFullDomains(Origin)
const Groups = new Map([...EachDomain].map(Group => [Group, MatchingGroups.get(GetDomainRootLabel(Group.values().next().value!))!] as const))
const EachDomainFull = new Set(Groups.values())
const Result: TASDomainContainer = new Map()
Result.set('Normal', NormalDomains)
Result.set('Full', FullDomains)
Result.set('EachDomain', EachDomain)
Result.set('EachDomainFull', EachDomainFull)
Result.set('Groups', Groups)
return Result
}

export async function FetchAdShieldDomains(): Promise<TASDomainContainer> {
Expand All @@ -54,16 +57,5 @@ export async function FetchAdShieldDomains(): Promise<TASDomainContainer> {
})
CustomDefinedMatches.forEach(Match => CombinedDomains.add(Match))
CustomExcludeMatches.forEach(Match => CombinedDomains.delete(Match))
const Result: TASDomainContainer = new Map()
const NormalDomains = DiscardResolvedDupWildcard(CombinedDomains)
const FullDomains = ConvertToFullDomains(NormalDomains)
const EachDomain = RegroupDomainTldLevel(NormalDomains)
const EachDomainFull = ConvertToFullDomains(EachDomain)

Result.set('Normal', NormalDomains)
Result.set('Full', FullDomains)
Result.set('EachDomain', EachDomain)
Result.set('EachDomainFull', EachDomainFull)

return Result
return CreateDomainViews(CombinedDomains)
}
26 changes: 12 additions & 14 deletions libs/source/references/utils/discard-resolved-dup-wildcard.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import * as TLDTS from 'tldts'
import { PublicSuffixList } from './wildcard-suffix-converter.js'

type ParsedEntry = {
Entry: string
Expand All @@ -23,9 +22,6 @@ function ParseEntry(Entry: string): ParsedEntry {
}

const Parsed = TLDTS.parse(Entry)
if (Parsed.publicSuffix) {
PublicSuffixList.add(Parsed.publicSuffix)
}

if (Parsed.hostname && Parsed.publicSuffix && Parsed.hostname.endsWith(`.${Parsed.publicSuffix}`)) {
return {
Expand All @@ -48,7 +44,10 @@ function ParseEntry(Entry: string): ParsedEntry {
}

function IsCoveredByParent(Child: ParsedEntry, Parent: ParsedEntry): boolean {
if (Child.Stem === Parent.Stem || !Child.Stem.endsWith(`.${Parent.Stem}`)) {
if (Child.Stem === Parent.Stem) {
return Parent.WildcardSuffix && !Child.WildcardSuffix
}
if (!Child.Stem.endsWith(`.${Parent.Stem}`)) {
return false
}

Expand All @@ -61,23 +60,22 @@ function IsCoveredByParent(Child: ParsedEntry, Parent: ParsedEntry): boolean {

export function DiscardResolvedDupWildcard(OriginSet: Set<string>): Set<string> {
const ParsedEntries = [...OriginSet].map(ParseEntry)
const RootsWithWildcard = new Set(
ParsedEntries
.filter(Entry => Entry.WildcardSuffix)
.map(Entry => Entry.RootLabel)
)

const ConcreteStemCounts = new Map<string, number>()
const ConcreteSuffixes = new Map<string, Set<string>>()
ParsedEntries
.filter(Entry => !Entry.WildcardSuffix)
.forEach(Entry => ConcreteStemCounts.set(Entry.Stem, (ConcreteStemCounts.get(Entry.Stem) ?? 0) + 1))
.filter(Entry => !Entry.WildcardSuffix && Entry.PublicSuffix)
.forEach(Entry => {
const Suffixes = ConcreteSuffixes.get(Entry.Stem) ?? new Set<string>()
Suffixes.add(Entry.PublicSuffix!)
ConcreteSuffixes.set(Entry.Stem, Suffixes)
})

const NormalizedEntries = ParsedEntries.map(Entry => {
if (Entry.WildcardSuffix) {
return Entry
}

if (RootsWithWildcard.has(Entry.RootLabel) || (ConcreteStemCounts.get(Entry.Stem) ?? 0) >= 2) {
if ((ConcreteSuffixes.get(Entry.Stem)?.size ?? 0) >= 2) {
return ParseEntry(`${Entry.Stem}.*`)
}

Expand Down
8 changes: 4 additions & 4 deletions libs/source/references/utils/regroup-domain-tldlevel.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import * as TLDTS from 'tldts'
import { PublicSuffixList } from './wildcard-suffix-converter.js'

type ParsedEntry = {
Entry: string
Expand All @@ -23,9 +22,6 @@ function ParseEntry(Entry: string): ParsedEntry {
}

const Parsed = TLDTS.parse(Entry)
if (Parsed.publicSuffix) {
PublicSuffixList.add(Parsed.publicSuffix)
}

if (Parsed.hostname && Parsed.publicSuffix && Parsed.hostname.endsWith(`.${Parsed.publicSuffix}`)) {
return {
Expand All @@ -47,6 +43,10 @@ function ParseEntry(Entry: string): ParsedEntry {
}
}

export function GetDomainRootLabel(Entry: string): string {
return ParseEntry(Entry).RootLabel
}

function IsCoveredByParent(Child: ParsedEntry, Parent: ParsedEntry): boolean {
if (Child.Stem === Parent.Stem || !Child.Stem.endsWith(`.${Parent.Stem}`)) {
return false
Expand Down
12 changes: 0 additions & 12 deletions libs/source/references/utils/wildcard-suffix-converter.ts

This file was deleted.

31 changes: 31 additions & 0 deletions libs/test/references/banner.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
import { expect, test } from 'vitest'
import { CreateBanner } from '../../../builder/source/banner/index.js'

test('banner uses match rules only for collected concrete domains', () => {
const Url = new URL('https://example.com/')
const Banner = CreateBanner({
Version: '1', BuildType: 'production', Domains: new Set(['example.com', 'example.net', 'wild.*']),
Author: 'author', Name: 'name', Namespace: 'namespace', HomepageURL: Url, SupportURL: Url,
UpdateURL: Url, DownloadURL: Url, License: 'license', Description: { en: 'description' }
})
expect(Banner.match(/^\/\/ @match\s+.*$/gm)).toEqual([
'// @match *://example.com/*',
'// @match *://*.example.com/*',
'// @match *://example.net/*',
'// @match *://*.example.net/*'
])
expect(Banner).not.toContain('@include')
expect(Banner).not.toContain('wild.*')
})

test('banner rejects an empty or wildcard-only matching set', () => {
const Url = new URL('https://example.com/')
const Options = {
Version: '1', BuildType: 'production' as const, Domains: new Set<string>(),
Author: 'author', Name: 'name', Namespace: 'namespace', HomepageURL: Url, SupportURL: Url,
UpdateURL: Url, DownloadURL: Url, License: 'license', Description: { en: 'description' }
}
expect(() => CreateBanner(Options)).toThrow('No concrete domains')
Options.Domains.add('wild.*')
expect(() => CreateBanner(Options)).toThrow('No concrete domains')
})
4 changes: 2 additions & 2 deletions libs/test/references/discard-resolved-dup-wildcard.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,14 @@ test('DiscardResolvedDupWildcard removes resolved duplicate wildcards with multi

test('DiscardResolvedDupWildcard handles nested wildcard scenarios', () => {
const Input = new Set(['token.google.*', 'access.google.*', 'tools.google.com', 'google.google.co.kr'])
const Expected = new Set(['token.google.*', 'access.google.*', 'tools.google.*', 'google.google.*'])
const Expected = Input

return expect(DiscardResolvedDupWildcard(Input)).toEqual(Expected)
})

test('DiscardResolvedDupWildcard handles complex wildcard scenarios', () => {
const Input = new Set(['token.google.*', 'access.google.*', 'tools.google.com', 'google.google.co.kr', 'example.*', 'example.com', 'rust-lang.org'])
const Expected = new Set(['token.google.*', 'access.google.*', 'tools.google.*', 'google.google.*', 'example.*', 'rust-lang.org'])
const Expected = new Set(['token.google.*', 'access.google.*', 'tools.google.com', 'google.google.co.kr', 'example.*', 'rust-lang.org'])

return expect(DiscardResolvedDupWildcard(Input)).toEqual(Expected)
})
Loading
Loading