Quick definitions (operational)
- temperature — scales how peaked or flat the next-token distribution is. Higher values usually make outputs more varied; lower values make them more concentrated.
- top_p — keeps only the smallest set of tokens whose cumulative probability reaches p, then samples within that set.
Exact behaviour depends on the model and provider. Treat docs as the source of truth for allowed ranges.
Watch: youtube.com/shorts/dBZDWJRRDGw.
Method: change one knob at a time
- Fix the prompt, system message, and model id.
- Run baseline: temperature and top_p at the provider defaults (or the values in your production config).
- Vary only temperature across 2–3 values; keep top_p fixed.
- Reset temperature; vary only top_p across 2–3 values.
- Save each raw completion with the parameter pair in the filename.
What to look at
- Does the answer still follow required format (JSON, bullet list, language)?
- Did factual claims stay stable across runs, or did names/numbers drift?
- For creative drafts, is the extra variety actually useful — or just longer?
Practical tip
For structured extraction or JSON, start low on temperature and leave top_p near the provider default until format is stable. For brainstorming, raise temperature gradually and stop when edits become more work than help. Avoid changing both knobs in the same experiment.
Choosing AI toolsPrompt basicsHow to try three free AI tools with your oOne API key, many models: how to evaluate System prompt mistakes: how to review and ← Back to home