Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/workflows/crawl_tj_all_number.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
name: Crawl All TJ Song By Number

on:
workflow_dispatch: # 수동 실행 전용

permissions:
contents: write

Comment on lines +6 to +8

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

4. Workflow has write permission 🐞 Bug ⛨ Security

The new crawl workflow requests contents: write even though it only checks out code and runs
scripts. This unnecessarily expands the blast radius if the workflow is ever compromised.
Agent Prompt
## Issue description
The workflow declares `permissions: contents: write` but does not appear to push commits/tags/releases. Granting write access is unnecessary and increases risk.

## Issue Context
Steps: checkout, setup node/pnpm, install deps, write `.env`, run `pnpm run tj-all-number`.

## Fix Focus Areas
- .github/workflows/crawl_tj_all_number.yml[6-8]

## Suggested fix
- Change to `permissions: contents: read` (or remove the block entirely to use the default minimal permissions), unless a later step truly requires write access.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

jobs:
run-npm-task:
runs-on: ubuntu-latest

# 6시간 잡 제한 안에서 전 대역을 커버하기 위해 1만 단위로 구간을 나눠 병렬 실행
strategy:
fail-fast: false
matrix:
range:
- { start: 1, end: 10000 }
- { start: 10001, end: 20000 }
- { start: 20001, end: 30000 }
- { start: 30001, end: 40000 }
- { start: 40001, end: 50000 }
- { start: 50001, end: 60000 }
- { start: 60001, end: 70000 }
- { start: 70001, end: 80000 }
- { start: 80001, end: 90000 }
- { start: 90001, end: 99999 }

name: "crawl ${{ matrix.range.start }}-${{ matrix.range.end }}"

steps:
- name: Checkout branch
uses: actions/checkout@v4

- name: Use Node.js 20
uses: actions/setup-node@v4
with:
node-version: "20"

- name: Install pnpm
uses: pnpm/action-setup@v2
with:
version: 9
run_install: false

- name: Install dependencies
working-directory: packages/crawling
run: pnpm install

- name: Create .env file
working-directory: packages/crawling
run: |
echo "SUPABASE_URL=${{ secrets.SUPABASE_URL }}" >> .env
echo "SUPABASE_KEY=${{ secrets.SUPABASE_KEY }}" >> .env

- name: run script - crawlAllTJSongByNumber.ts
working-directory: packages/crawling
env:
START_NUMBER: ${{ matrix.range.start }}
END_NUMBER: ${{ matrix.range.end }}
run: pnpm run tj-all-number
40 changes: 0 additions & 40 deletions .github/workflows/crawl_tj_by_number.yml

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/tagging_song.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Tagging Songs

on:
schedule:
- cron: "0 10 * * 1" # 매주 월요일 한국 시간 19:00 실행 (UTC+9 → UTC 10:00)
- cron: "0 7 * * *" # 매일 한국 시간 16:00 실행 (UTC+9 → UTC 7:00)
workflow_dispatch:

permissions:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/translation_jpn.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Translation J-POP Songs

on:
schedule:
- cron: "0 10 * * 5" # 매주 금요일 한국 시간 19:00 실행 (UTC+9 → UTC 10:00)
- cron: "0 10 * * *" # 매일 한국 시간 19:00 실행 (UTC+9 → UTC 10:00)
workflow_dispatch:

permissions:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/verify_ky_youtube.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Verify ky by Youtube

on:
schedule:
- cron: "0 14 * * *" # 한국 시간 23:00 실행 (UTC+9 → UTC 14:00)
- cron: "0 14 * * 1" # 매주 월요일 한국 시간 23:00 실행 (UTC+9 → UTC 14:00)
workflow_dispatch:

permissions:
Expand Down
18 changes: 18 additions & 0 deletions apps/twa/.gitignore
Original file line number Diff line number Diff line change
@@ -1,9 +1,27 @@
# Android keystore — 외부 백업 필수, git에 절대 커밋하지 않는다
*.keystore
*.jks
*.keystore.bak*
android.keystore.bak*

# Bubblewrap 빌드 산출물
android/
output/
app/
*.aab
*.apk
*.apk.idsig

# Gradle 생성물
.gradle/
gradle/
gradlew
gradlew.bat
build.gradle
settings.gradle
gradle.properties
manifest-checksum.txt
store_icon.png

# 수동 백업/복사본
twa-manifest copy.json
40 changes: 21 additions & 19 deletions apps/twa/twa-manifest.json
Original file line number Diff line number Diff line change
@@ -1,16 +1,17 @@
{
"packageId": "kr.singcode.app",
"host": "singcode.kr",
"name": "Singcode - 당신의 노래방 메모장",
"name": "싱코드(singcode) - 노래방 번호 검색",
"launcherName": "Singcode",
"display": "standalone",
"themeColor": "#1a1a2e",
"navigationColor": "#1a1a2e",
"navigationColorDark": "#1a1a2e",
"navigationDividerColor": "#1a1a2e",
"navigationDividerColorDark": "#1a1a2e",
"backgroundColor": "#1a1a2e",
"enableNotifications": false,
"themeColor": "#1A1A2E",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. themecolor not #1a1a2e 📘 Rule violation ≡ Correctness

apps/twa/twa-manifest.json sets themeColor to #1A1A2E instead of the required lowercase
#1a1a2e. This can break enforcement checks that expect the exact string and violates the mandated
theme color value.
Agent Prompt
## Issue description
`twa-manifest.json` must enforce `themeColor` exactly as `"#1a1a2e"` (lowercase), but the PR sets it to `"#1A1A2E"`.

## Issue Context
Compliance rule requires an exact string match for the TWA manifest theme color.

## Fix Focus Areas
- apps/twa/twa-manifest.json[7-7]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

"themeColorDark": "#1A1A2E",
"navigationColor": "#1A1A2E",
"navigationColorDark": "#1A1A2E",
"navigationDividerColor": "#1A1A2E",
"navigationDividerColorDark": "#1A1A2E",
"backgroundColor": "#1A1A2E",
"enableNotifications": true,
"startUrl": "/",
"iconUrl": "https://singcode.kr/icons/icon-512.png",
"maskableIconUrl": "https://singcode.kr/icons/icon-maskable-512.png",
Expand All @@ -20,20 +21,13 @@
"path": "./android.keystore",
"alias": "singcode"
},
"appVersionCode": 1,
"appVersionName": "1.0.0",
"appVersionName": "3",
"appVersionCode": 3,
"shortcuts": [],
"generatorApp": "bubblewrap-cli",
"webManifestUrl": "https://singcode.kr/manifest.webmanifest",
"fallbackType": "customtabs",
"features": {
"locationDelegation": {
"enabled": false
},
"playBilling": {
"enabled": false
}
},
"features": {},
"alphaDependencies": {
"enabled": false
},
Expand All @@ -42,5 +36,13 @@
"isMetaQuest": false,
"fullScopeUrl": "https://singcode.kr/",
"minSdkVersion": 21,
"orientation": "portrait"
"orientation": "portrait",
"fingerprints": [],
"additionalTrustedOrigins": [],
"retainedBundles": [],
"protocolHandlers": [],
"fileHandlers": [],
"launchHandlerClientMode": "",
"displayOverride": [],
"appVersion": "3"
}
3 changes: 2 additions & 1 deletion apps/web/public/.well-known/assetlinks.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
"namespace": "android_app",
"package_name": "kr.singcode.app",
"sha256_cert_fingerprints": [
"4A:52:E2:11:4D:E0:6F:B6:E1:E1:B2:BB:72:14:8C:0B:17:97:06:DA:BD:F7:DB:A2:3B:10:59:46:CD:79:3B:E9"
"4A:52:E2:11:4D:E0:6F:B6:E1:E1:B2:BB:72:14:8C:0B:17:97:06:DA:BD:F7:DB:A2:3B:10:59:46:CD:79:3B:E9",
"06:63:E8:E2:C6:97:EE:CE:BF:7C:2D:24:2B:F0:14:61:3F:01:39:4D:8E:B2:B3:E0:F4:65:45:E9:C1:F3:C4:0E"
]
}
}
Expand Down
9 changes: 4 additions & 5 deletions apps/web/src/app/layout.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,11 @@ import QueryProvider from '@/query';
const isDevelopment = process.env.NODE_ENV === 'development';

export const metadata: Metadata = {
title: 'Singcode - 당신의 노래방 메모장',
description:
'노래방만 가면 부르고 싶었던 노래가 기억 안 날 때? Singcode에서 검색하고 저장하면 걱정 끝!',
title: 'Singcode - 노래방 번호 검색',
description: 'Singcode에서 빠르고 편하게 노래방 번호 검색하세요. J-POP 검색도 지원합니다.',
openGraph: {
title: 'Singcode - 노래방에서 부를 곡, 기억하지 말고 저장하세요',
description: '노래방 갈 때마다 잊어버리는 곡번호? Singcode가 대신 기억할게요!',
title: 'Singcode - 노래방 번호 검색',
description: 'Singcode에서 빠르고 편하게 노래방 번호 검색하세요. J-POP 검색도 지원합니다.',
url: 'https://www.singcode.kr',
siteName: 'Singcode',
images: [
Expand Down
1 change: 1 addition & 0 deletions packages/crawling/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
"recent-tj": "tsx src/cron/crawlRecentTJ.ts",
"tag-songs": "tsx src/cron/taggingSongs.ts",
"trans-jpn": "tsx src/cron/translationJpn.ts",
"tj-all-number": "tsx src/cron/crawlAllTJSongByNumber.ts",
"lint": "eslint .",
"test": "vitest run",
"format": "prettier --write \"**/*.{ts,tsx,md}\""
Expand Down
6 changes: 4 additions & 2 deletions packages/crawling/src/crawling/crawlTJSongByNumber.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,10 @@ export const crawlTJSongByNumber = async (

const gridContainer = $('.grid-container.list.ico').first();

const title = gridContainer.find('.grid-item.title3').text().trim();
const artist = gridContainer.find('.grid-item.title4').text().trim();
// title3 내부에는 아이콘용 <p class="no-ico">가 div.flex-box > ul 안에 중첩돼 있으므로
// 자손 셀렉터(' p') 대신 직계 자식(' > p')만 선택해 실제 제목만 추출한다.
const title = gridContainer.find('.grid-item.title3 span').text().trim();
const artist = gridContainer.find('.grid-item.title4 span').text().trim();
Comment on lines +25 to +28

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Informational

5. Misleading tj selector comment 🐞 Bug ⚙ Maintainability

The comment says to use a direct-child > p selector to avoid nested icon markup, but the
implementation selects span elements instead. This mismatch makes future maintenance risky because
the rationale and the actual selector no longer align.
Agent Prompt
## Issue description
`crawlTJSongByNumber()` includes a comment describing a `> p` direct-child strategy, but the code uses `.find('... span')`. This discrepancy is confusing and can cause incorrect future edits.

## Issue Context
A separate TJ crawler parses title text from nested `<p>` elements, so the exact DOM structure is subtle and should be documented accurately.

## Fix Focus Areas
- packages/crawling/src/crawling/crawlTJSongByNumber.ts[25-28]

## Suggested fix
- Either update the comment to match the current `span`-based extraction (and explain why `span` is correct), or update the selector to match the documented `> p` approach.
- Consider adding a small unit/regression test with a saved HTML fixture to lock the intended extraction behavior.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


if (!title || !artist) {
console.log('❌ TJ 검색 결과 없음');
Expand Down
17 changes: 14 additions & 3 deletions packages/crawling/src/cron/crawlAllTJSongByNumber.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,20 @@ import { Song } from '@/types';

dotenv.config();

// 순회 범위
const START_NUMBER = 1;
const END_NUMBER = 99999;
// 순회 범위 (env로 주입, 미설정 시 전체 범위)
const parseRange = (value: string | undefined, fallback: number): number => {
const parsed = Number(value);
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
};

const START_NUMBER = parseRange(process.env.START_NUMBER, 1);
const END_NUMBER = parseRange(process.env.END_NUMBER, 99999);

if (START_NUMBER > END_NUMBER) {
throw new Error(`잘못된 범위: START_NUMBER(${START_NUMBER}) > END_NUMBER(${END_NUMBER})`);
}

console.log(`🔢 순회 범위: ${START_NUMBER} ~ ${END_NUMBER}`);

// 1. 최초로 DB의 모든 곡 데이터를 받아서 num_tj 기준 Map으로 보관
const allSongs = await getSongsAllWithTjDB();
Expand Down
12 changes: 6 additions & 6 deletions packages/crawling/src/supabase/getDB.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ export async function getSongsJpnDB() {
return hasJapaneseData;
}

export async function getSongsKyNullDB(max: number = 50000) {
export async function getSongsKyNullDB(max: number = 100000) {
const supabase = getClient();

const { data, error } = await supabase
Expand All @@ -48,7 +48,7 @@ export async function getSongsKyNullDB(max: number = 50000) {
return data;
}

export async function getSongsKyNotNullDB(max: number = 50000) {
export async function getSongsKyNotNullDB(max: number = 100000) {
const supabase = getClient();

const { data, error } = await supabase
Comment on lines +51 to 54

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

3. Excessive supabase overfetch 🐞 Bug ➹ Performance

The default max for several DB fetch functions was increased to 100,000, but callers often process
only a small prefix (e.g., verify job stops at 5,000; tagging stops at 20,000). This creates
unnecessary data transfer and memory usage each run.
Agent Prompt
## Issue description
`getSongsKyNotNullDB()` (and other helpers) now default to `max=100000`, but some cron scripts intentionally stop after processing a much smaller number of rows. Fetching far more rows than needed increases bandwidth/memory and can slow down scheduled runs.

## Issue Context
- `crawlYoutubeVerify.ts` breaks after 5,000 items but currently calls `getSongsKyNotNullDB()` with default max.
- `taggingSongs.ts` breaks after 20,000 items but calls `getSongsAllDB()` with default max.

## Fix Focus Areas
- packages/crawling/src/supabase/getDB.ts[36-64]
- packages/crawling/src/supabase/getDB.ts[88-100]
- packages/crawling/src/cron/crawlYoutubeVerify.ts[19-49]
- packages/crawling/src/cron/taggingSongs.ts[11-24]

## Suggested fix
- Pass explicit caps from callers matching their processing limits (e.g., `getSongsKyNotNullDB(5000)` and `getSongsAllDB(20000)`).
- Alternatively, change defaults back to smaller values and require explicit `max` for “full table” use-cases (paired with pagination from the other finding).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Expand Down Expand Up @@ -85,7 +85,7 @@ export async function getVerifyKySongsDB(): Promise<Set<string>> {
return new Set(data.map(row => row.id));
}

export async function getSongsAllDB(max: number = 50000) {
export async function getSongsAllDB(max: number = 100000) {
const supabase = getClient();

const { data, error } = await supabase
Comment on lines +88 to 91

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

2. Db queries truncate silently 🐞 Bug ≡ Correctness

Several getDB() helpers use .limit(100000) without pagination, but callers treat the results as
complete (e.g., building num_tj maps or “already tagged” sets). Once tables exceed 100k rows, this
silently breaks dedupe/skip logic and can lead to duplicate inserts or redundant processing.
Agent Prompt
## Issue description
Supabase fetch helpers (e.g., `getSongsAllDB`, `getSongsAllWithTjDB`, `getSongTagSongIdsDB`) hard-cap results via `.limit(100000)` and return only the first page. Downstream jobs assume these are complete datasets for deduplication/skip decisions, so once any underlying table grows beyond the limit the jobs will silently behave incorrectly (re-tag already-tagged songs, miss existing TJ numbers and attempt inserts, etc.).

## Issue Context
- `taggingSongs.ts` loads *all songs* and *all tagged song IDs* to skip work.
- `crawlAllTJSongByNumber.ts` loads *all songs with num_tj* to decide whether to insert/update.
- Current fetch helpers do not implement pagination (`.range(...)`) or any loop until exhaustion.

## Fix Focus Areas
- packages/crawling/src/supabase/getDB.ts[88-115]
- packages/crawling/src/supabase/getDB.ts[155-163]
- packages/crawling/src/cron/taggingSongs.ts[11-28]
- packages/crawling/src/cron/crawlAllTJSongByNumber.ts[27-65]

## Suggested fix
1. Implement pagination utilities in `getDB.ts` (e.g., fetch in pages of 1000/5000 using `.range(offset, offset+pageSize-1)` until a page returns fewer than `pageSize`).
2. Update `getSongsAllWithTjDB()` and `getSongTagSongIdsDB()` to use pagination and return complete results (or, if “complete” is not required, rename functions to reflect partial reads and adjust callers accordingly).
3. Consider filtering server-side to reduce result size (e.g., `not('num_tj','is',null)` for the TJ map case).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Expand Down Expand Up @@ -121,7 +121,7 @@ export async function getJpopSongsForTranslationDB() {
.from('songs')
.select('id, title, artist, title_ko, artist_ko, song_tags!inner(tag_id)')
.eq('song_tags.tag_id', 101)
.limit(50000);
.limit(100000);

if (error) throw error;

Expand All @@ -138,7 +138,7 @@ export async function getArtistKoMapDB(): Promise<Map<string, string>> {
.select('artist, artist_ko, song_tags!inner(tag_id)')
.eq('song_tags.tag_id', 101)
.not('artist_ko', 'is', null)
.limit(50000);
.limit(100000);

if (error) throw error;

Expand All @@ -155,7 +155,7 @@ export async function getArtistKoMapDB(): Promise<Map<string, string>> {
export async function getSongTagSongIdsDB(): Promise<Set<string>> {
const supabase = getClient();

const { data, error } = await supabase.from('song_tags').select('song_id').limit(50000);
const { data, error } = await supabase.from('song_tags').select('song_id').limit(100000);

if (error) throw error;

Expand Down
Loading