27 May 2024 | Director Meeting

Scrapping

1. How to get the data: Reddit, Uninotes

How to Search Pages?

• Search each course code

• Use LLM?

2. Clean data:

• Discuss with the group

• Regex

3. Store data

• Store reviews in text file first

• Then use another program to consume text file to database

• Use Python

Reddit Scrapping Checkpoints

• Separate User & Reddit Review

  • Add an additional attribute

Wrapped

• Designs

• API Routes to process data

  • Put in text file or just use get calls?

  • Make sure API routes is reproducible

• Put wrapped title in navbar which links to another page

• Keep in the component files

• Tracking stats:

  • SQL filter on the last year

• Analytics:

  • How many views, engagements, etc