Please provide feedback in https://discord.com/invite/adh94jZH37 on which dimension(s) that you want to scorer to be improved on. We should be able to incorporate these improvements on the generic scorer before going into training a specific scorer for domain specific use-cases.
willtholke 329 days ago [-]
great looking ui and implementation, and kudos on the launch. how do you handle cases where predefined scoring metrics don’t fully capture what ‘good’ means for a specific use case, like ranking legal documents or detecting nuanced sentiment in customer reviews?
davidkara 320 days ago [-]
hey will, the metrics in the scoring system are not predefined but rather generated for a use case by breaking down a subjective set of conditions into a tree of metrics that combine various objective metrics into a subjective aggregate. Here is a prefilled playground with dimensions for the sentiment analysis in customer reviews use case. You can see how it breaks down and if you put a review and click Run you should see the scores combine from individual custom dimensions to the aggregate score. Hope this helps!
What kind of fine-tuning capabilities does the user have on the scorer? Would I be able focus on improving certain scorer dimensions over others?
Please provide feedback in https://discord.com/invite/adh94jZH37 on which dimension(s) that you want to scorer to be improved on. We should be able to incorporate these improvements on the generic scorer before going into training a specific scorer for domain specific use-cases.
https://build.withpi.ai/shared/021c9990-c02a-477c-8e32-fd2d5...