Parton-Level Cross Section Interpolation
Interpolate the relationship between a (parametrised) underlying distribution of particles and the probability of such an event occurring in a collider.
Hosted by
CERN
Posted
Question on dsigma and scoring
Hi,
I tried
aqora test for this challenge, and the score exploded to an extremely large number.I tested with the
events-validation.hdf5 file. Correct me if I'm wrong, the d_sigma in the file is the one used for scoring when running aqora test, and the scoring method is using MAPE. I calculated the MAPE score from my generated predictions with the events-validation.hdf5 data, and it is the same as the aqora test score, i.e. an extremely large value.I found out that the
d_sigma in the events-validation.hdf5 has many rows with very small values and many negative values. For instance, there are 11067 in the file with d_sigma ≤ 0. These negative values are exploding the MAPE.My question is, what is the physically possible minimum value for the
d_sigma? Can it be negative or close to 0?During training, I am filtering out all events with
d_sigma ≤ 0 in the training dataset since from what I understand it is a probability and cannot be negative?Thank you.
Order by:
2
Posted by eno •
But regarding the negativity of
d_sigma, yes, that can happen, even though we would usually regard this value as a probability for the event to happen. But it's a bit more complicated than this in practice. The negative values here come from the so-called parton density function, which can become negative (at higher orders in perturbation theory). Note that this is only a statement about individual points in the sample. Overall, with enough statistics, you will always get a positive sigma for any meaningful physics observable.1
Posted by grossim •
Hi there, apologise for this inconvenience. We do understand this, the problem is the higher parton density functions that can cause 'd_sigma' to become negative.
We propose the following: take the absolute value of this to calculate the MAPE score.
1
Posted by jag •
Hi @grossim, no worries. When you say take the absolute value of this, do you mean taking the absolute value of all
d_sigma in my predictions and the validation file and then computing the MAPE score based on that?1
Posted by grossim •
Yes, I think this should work.
1
Posted by jag • (edited)
@grossim I've trained my algorithm and these are the results I'm getting
RMSE: 142.9157
MAE: 60.2217
MAPE: 951165619279107584.00%
R²: 0.9240
The R² score seems good but the MAPE is still very large.
I'm computing the metrics as follows:
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error, mean_absolute_percentage_error, r2_score
mse = mean_squared_error(y_targets, y_preds)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_targets, y_preds)
mape = mean_absolute_percentage_error(np.abs(y_targets), np.abs(y_preds))
r2 = r2_score(y_targets, y_preds)
print(f"RMSE: {rmse:.4f}")
print(f"MAE: {mae:.4f}")
print(f"MAPE: {mape:.2%}")
print(f"R²: {r2:.4f}")I think R² could be a better metric to gauge the algorithm. R² ranges from 0 to 1, where 1 indicates a perfect fit of the algorithm to the data. On the other hand, MAPE seems to be giving these large numbers.