StackGP.generateRandomBenchmark

StackGP.generateRandomBenchmark#

StackGP.generateRandomBenchmark(numVars=5, numSamples=100, noiseLevel=0, opsChoices=defaultOps(), constChoices=defaultConst(), maxLength=10)

generateRandomBenchmark is a StackGP function that generates a random benchmark set which includes an input data set, a response vector, and a target model. This is useful when benchmarking modeling strategies to get a new benchmark problem. This help avoid the risk of overengineering for a specific benchmark problem.

The function has no required arguments

and has 6 optional arguments, numVars, numSamples, noiseLevel, opsChoices, constChoices, and maxLength

The arguments are described below:

  • numVars: The number of variables to include in the data. The model may not use all of them which allows us to test the feature selection ability of a ML method. The default setting is numVars=5.

  • numSamples: The number of data points to generate in the dataset. The default setting is numSamples=100.

  • noiseLevel: The amount of noise to include in the response vector. Useful in testing how well a ML method handles noise. The default set is noiseLevel=0.

  • opsChoices: The math operators allowed to be used in the model that generates the data. The default setting is opsChoices=defaultOps().

  • constChoices: The numeric constants allowed to be used in the model taht generates the data. The default setting is constChoices=defaultConst().

  • maxLength: The max stack length of the model used to generate the data. Longer models will be harder to rediscover. The default setting is maxLength=10.

First we need to load in the necessary packages

import StackGP as sgp

Overview#

Generate a random benchmark#

Here we generate a random benchmark problem with 5 features and 100 data points.

inputData, response, targetModel = sgp.generateRandomBenchmark(numVars=5, numSamples=100)

We can display the random model below

sgp.printGPModel(targetModel)
\[\displaystyle 3.14159265358979 x_{1} \sqrt{\frac{x_{2}^{2}}{x_{1}^{2}}}\]

We can preview the first 10 samples from inputData

inputData[:, :10]

array([[0.02704064, 0.460551  , 0.9658211 , 0.31174459, 0.42499308,
        0.32335141, 0.89328799, 0.36008447, 0.91338106, 0.56291424],
       [0.92080167, 0.40304422, 0.32421934, 0.96437513, 0.78243877,
        0.40542678, 0.23620553, 0.30865224, 0.21491979, 0.17771826],
       [0.53263347, 0.41001997, 0.45825104, 0.79629268, 0.03863028,
        0.38847829, 0.0972199 , 0.87640376, 0.56024455, 0.97048466],
       [0.21640761, 0.20408726, 0.21532482, 0.28872733, 0.66409168,
        0.33698185, 0.3768071 , 0.75977699, 0.42925892, 0.8338576 ],
       [0.09197239, 0.95683331, 0.28734927, 0.99007371, 0.24997691,
        0.4816863 , 0.38356994, 0.73582091, 0.3927366 , 0.66306159]])

We can preview a plot of the response vector.

import matplotlib.pyplot as plt
plt.plot(response, label='Target Response', color='blue', marker="o")
plt.xlabel('Sample Index')
plt.ylabel('Response')
plt.title('Target Response vs Sample Index')
plt.legend()
plt.show()
../_images/8a3147b8edeb679e8fd3537223db13dd4f7480ca19a9d8764972f57a4ffdb267.png

Now we can test the ability for StackGP to rediscover the target model using the benchmark data.

models = sgp.parallelEvolve(inputData,response,tracking=True)

Running parallel evolution with 16 jobs.
../_images/cb7a459ed6cae5c3957727812a179521ee14f483361adf70d94eb29f0240dfa2.png

We can plot the model population to see the quality of the population. It looks like there are some models that are very simple and accurate.

sgp.plotModels(models)
../_images/1500eb0f23d942e70f025cdd339a29c89bbfd04bb83c61c09320d639a5ba44bd.png

Now we can check the symbolic form of the best model to see if it matches the generating function.

sgp.printGPModel(models[0])
\[\displaystyle 3.1415926535898 x_{2}\]

Options#

This section showcases how each of the different arguments can be used with the generateRandomBenchmark function.

numVars: Controlling number of variables#

We can change the number of features in the data set and the max number of features allowed in the model using numVars.

In this example lets decrease the number of variables down to 1.

inputData, response, targetModel = sgp.generateRandomBenchmark(numVars=1)
sgp.printGPModel(targetModel)
\[\displaystyle 3.14159265358979 e^{- x_{0}}\]

We could also greatly increase the number of variables. For example to 50.

inputData, response, targetModel = sgp.generateRandomBenchmark(numVars=50)
sgp.printGPModel(targetModel)
\[\displaystyle \frac{1}{x_{34} \left(\frac{x_{2} x_{3}}{x_{31}} - x_{37}\right)}\]

If we want to increase the number of variables in use in the model while increasing the number of variables in the data, we would also need to increase the max allowed complexity of the model. For example, here we increase the number of variables to 50 and the max size of the model to 30.

inputData, response, targetModel = sgp.generateRandomBenchmark(numVars=50, maxLength=30)
sgp.printGPModel(targetModel)
\[\displaystyle x_{33} + \left(x_{3} + x_{36}\right)^{2} \left(x_{20} + x_{21} + \sqrt{x_{28} \left(x_{12} + x_{17}\right)}\right)^{2} - \frac{1}{x_{38}}\]

This makes for a very challenging benchmark problem.

models = sgp.parallelEvolve(inputData,response,tracking=True)

Running parallel evolution with 16 jobs.
../_images/f2d8a31c7166c360f9fb2c5dd6b0993d353037752d5518f4a66a4b10c887ce6f.png 
sgp.printGPModel(models[0])
\[\displaystyle 0.998542365165047 \left(x_{20} + \left(x_{28} x_{3} + x_{36}\right)^{2}\right)^{2} + 1.89526541716437 - 0.998542365165047 e^{- 2 e^{x_{38}}} - \frac{0.998542365165047}{x_{38}}\]

numSamples: Controlling Number of Data Samples#

We may want to change the number of data samples in the benchmark. Sometimes we may want very small datasets to see if our ML method can efficiently utilize the data. Sometime we want to use large datasets to test the efficiency of using a lot of data. Other times, a problem may be so challenging that we just need a ton of data.

In this example below we decrease the number of samples to just 3 points and then try to find a model with those points.

inputData, response, targetModel = sgp.generateRandomBenchmark(numSamples=3)
sgp.printGPModel(targetModel)
\[\displaystyle \frac{0.0822756643004843}{- x_{0} + x_{3} + 3.14159265358979}\]
# train models
models = sgp.parallelEvolve(inputData,response,tracking=True)

Running parallel evolution with 16 jobs.
../_images/6eed1bd339e02626cb54f2039abac1cf12d932fc883d73d57b64f4c0e63680b4.png 
# plot models
sgp.plotModels(models)
../_images/3f57a5305eca014bfae653b9c8876989bf96eb336ef2a268b0ef225e01c7fdde.png 
# print the best model found
sgp.printGPModel(models[0])
\[\displaystyle -6.00925839494864 \cdot 10^{-18} - \frac{0.0822756643004843}{x_{0} - x_{3} - 3.14159265358979}\]
# plot the model prediction vs the target response
sgp.plotModelResponseComparison(models[0], inputData, response)
../_images/8ce61017e242b9968d155d414691fd60ffd708fe4fe43d175d1b058e9df2ff96.png

Here we increase the data set size to 10,000 to test the efficiency of our balancedSample strategy.

inputData, response, targetModel = sgp.generateRandomBenchmark(numSamples=10000)
sgp.printGPModel(targetModel)
\[\displaystyle 646.140541305599 \sqrt{2.39522335484306 \cdot 10^{-6} x_{1} + e^{x_{1}}}\]
models = sgp.evolve(inputData, response, liveTracking=True, liveTrackingInterval=0, dataSubsample=True, samplingMethod=sgp.balancedSample)
../_images/33a84fefd6966c562303b850d1d573ced31b50adc25c1203441f7c66d1121b9f.png 
sgp.printGPModel(models[0])
\[\displaystyle 646.141251044859 \sqrt{e^{x_{1}}} - 0.000641622956663923\]

noiseLevel: Controlling amount of noise.#

We may want to inject noise into the benchmark data to make it more challenging to rediscover the benchmark model as well as to assess the ML method’s sensitivity to noise. We can do this by modifying the noiseLevel argument.

In this example we increase the noise level to 10.

inputData, response, targetModel = sgp.generateRandomBenchmark(noiseLevel=0.1)
sgp.printGPModel(targetModel)
\[\displaystyle x_{0} \sqrt{e^{2.71828182845905 x_{1}}}\]

Now we can observe the difference between the truth model output and the noise corrupted output generated.

import matplotlib.pyplot as plt
truthResponse = sgp.evaluateGPModel(targetModel, inputData)
plt.plot(truthResponse, label='Target Response', color='blue', marker="o")
plt.plot(response, label='Noisy Response', color='orange', marker="x")
plt.xlabel('Sample Index')
plt.ylabel('Response')
plt.title('Target Response vs Noisy Response')
plt.legend()
plt.show()
../_images/e5e749759df0628572a551c6deb8b64869c270457af3578b137b9785c11689e1.png 
models = sgp.parallelEvolve(inputData,response,tracking=True)

Running parallel evolution with 16 jobs.
../_images/6e745c2ce2a09654d75ba080150482f970cc6e4d0586c47d0be81431d71e99a8.png 
sgp.printGPModel(models[0])
\[\displaystyle 32.7984216425862 - 32.8650971528832 e^{- 0.0147539262939216 x_{0} \left(x_{1} + 2.13655935188817\right) e^{x_{1}}}\]
predictedResponse = sgp.evaluateGPModel(models[0], inputData)
plt.plot(truthResponse, label='Target Response', color='blue', marker="o")
plt.plot(predictedResponse, label='Model Prediction', color='green', marker="x")
plt.plot(response, label='Noisy Response', color='orange', marker=".")
plt.xlabel('Sample Index')
plt.ylabel('Response')
plt.title('Target Response vs Model Prediction')
plt.legend()
plt.show()
../_images/a8ad99e27d6e82a00e4dce4fb332fffb0a410d947b2bfe0dafd361a391a8421e.png

opsChoices#

We may want to control the operators available in the model by either controlling for a specific set or expanding to a broader set to make search more difficult.

In this first example, we make the set more specific to just addition and subtraction.

inputData, response, targetModel = sgp.generateRandomBenchmark(opsChoices=[sgp.add, sgp.sub])
sgp.printGPModel(targetModel)
\[\displaystyle - 2 x_{2} - 2 x_{3} + x_{4} + 6.85987448204884\]

Maybe we want to expand to the full operator set in StackGP using sgp.allOps()

inputData, response, targetModel = sgp.generateRandomBenchmark(opsChoices=sgp.allOps())
sgp.printGPModel(targetModel)
\[\displaystyle \tan{\left(\cos^{2}{\left(x_{4} \right)} + 22.4811843754167 \right)}\]

Maybe we are interested in benchmarking a system with very specific operators. Here we generate a benchmark with just +, *, and sin.

inputData, response, targetModel = sgp.generateRandomBenchmark(opsChoices=[sgp.add, sgp.mult, sgp.sin])
sgp.printGPModel(targetModel)
\[\displaystyle \sin{\left(x_{4}^{2} \sin{\left(29.1929245956293 x_{0} x_{2} \right)} + 5.85987448204884 \right)}\]

constChoices#

We can supply a list of specific constants if we know that a specific set of constants will be useful. Note, that other constants can appear via combinations of the constants in the supplied set.

inputData, response, targetModel = sgp.generateRandomBenchmark(constChoices=[1,5,10])
sgp.printGPModel(targetModel)
\[\displaystyle \frac{1}{x_{4}^{2} - 5}\]

Maybe we are interested in just having integer values

inputData, response, targetModel = sgp.generateRandomBenchmark(constChoices=[sgp.randomInt])
sgp.printGPModel(targetModel)
\[\displaystyle x_{0} \left(2 + \frac{1}{2 x_{4}}\right)\]

maxLength#

To make more complex problems we can increase the maxLength. To make a simpler problem, we can make smaller models using a smaller maxLength.

In the first example we decrease the length to 1, which means a model can have at most 1 operator.

inputData, response, targetModel = sgp.generateRandomBenchmark(maxLength=1)
sgp.printGPModel(targetModel)
\[\displaystyle e^{x_{4}}\]

We can drastically increase this value to get much more complicated models. Here we increase it to 100. Note: maxLength is a maximum value, so simpler model can still be returned.

inputData, response, targetModel = sgp.generateRandomBenchmark(maxLength=100)
sgp.printGPModel(targetModel)
\[\displaystyle \sqrt{\left(- 15.8412403029305 x_{0}^{2} \left(0.0138728021738592 - 1.05043227046255 \cdot 10^{-10} \sqrt{x_{0} x_{2} \left(0.037037037037037 x_{2}^{2} - e^{x_{3}}\right)^{2}}\right) + 15.8412403029305 x_{1}\right) \left(- \frac{\sqrt{3} x_{3}}{3} - 0.0603261248923995 x_{4} - 21.9721581335718\right)}\]

Examples#

This section some neat examples using the generateRandomBenchmark function.