Research Methods in Computer Science

How do we know stuff? How do we know something with enough certainty to make estimates about the future? You have an idea for solving a problem or creating a new service, a startup, and a finite amount of investor money and time to get something done; how do you decide what to do, and what not to do?

The process of producing knowledge (and appropriately confident estimates of possible futures and the expected oucomes of possible interventions) is the process of research. The academic sphere has accumulated lots of best practices in how to research well, and is continually investing in improving them. So does industry -- ever heard of A/B testing? -- but their specifics are often secret, and they anyway very much rely on academia to get their methods for producing actionable knowledge right.

Research questions range from "big science", such as "How old is the universe?" (kind of known) or "How does the evolution of culture work?" (very much not known), to very pragmatic: "Is it worth training a transformer for this task over a simple support vector machine?" "Is it worth paying for a higher-tier AI subscription, or can I solve this well enough with a bunch of heuristics? Or open-weight models?" "Does this change to the front-end lead to a higher click-through rate for ads?"

This subject should equip you to perform research with proper methodology and communicate your results convincingly. It will be most immediately useful for your bachelor thesis, but also it should equip you for any work where evidence-based decision making is needed. We will focus mostly on how this is done in computer science, especially machine learning and natural language processing. The subject covers formulating research questions, choosing evaluation metrics, designing test datasets and surveys, as well as ethical questions and working with human subjects. We also discuss why to engage with scientific literature, how to do it both as a writer and a reviewer, and we also discuss the possible roles of AI in the research process.

All lectures are accompanied by practical lab work (approx. 1:1).

Learning outcomes

A successful student of NPFL154:

• Knows how to design and carry out a simple experiment (e.g., for the purpose of evaluating their ideas in a bachelor thesis)
• Has a basic understanding of ethical aspects of research (both human subjects and data)
• Has an overview of different kinds of research: experimental, observational, qualitative, etc.
• Understands how scientific literature is produced and evaluated, and can perform basic literature review

Evaluation

"Zápočet" for active participation in labs.