Recap part 1 #

In part 1, I created some multi-objective optimisation models with OPL using both the CP Optimiser and CPLEX Optimiser. However, I hit a no-documentation barrier when started using the staticLexFull() function and found the DOcplex.mp Python example to be a more readable.

In this post, I’ll take a look at first converting the existing weighted model to Python, proceed with adding additional weights to the model and finally deploy the model to run on IBM Cloud with Watson Machine Learning. In part one, I did mention a full GUI but at this point but after some considerations, there’s no use case for a UI at the moment.

What’s DOcplex? #

DOcplex stands for Decision Optimisation CPLEX. It’s a Python library composed of two modules:

  • docplex.mp: taps into CPLEX Optimiser APIs, mp stands for mathematical programming
  • docplex.cp: taps into CP Optimiser APIs, cp stands for constraint programming

The library is advertised to be compatible with panda and numpy. The official documentation for this library is located at IBM Decision Optimisation Github Page. The documentation looks like it was written for someone who’s already been using CPLEX, i.e. with a lot of assumed knowledge. Aside from the setup intructions, it goes straight to the examples, and there’s only one multi-opjective optimisation example using the MP variant.

As of January 4th 2021, docplex library package that comes with CPLEX Optimisation Studio v20.1.0 relies on a legacy cplex library that only works with Python 3.8 and below.

Code sample #

The complete code listing for this post will be available in https://github.com/rampadc/multi-obj-cplex-python.

Journey to Pythonic CPLEX #

The Python code will make heavy use of Python’s List Comprehension feature.

Shout out to Nick Renotte’s video Solving Optimization Problems with Python Linear Programming. It really helped me when I got stuck at converting the sum() from OPL to Python.

Rewrite the OPL model in Python #

In this exercise, I’m using Jupyter Notebook to learn docplex. GitHub can format a IPython notebook much better than Hugo can. To see the result, visit 01-minimize-staticLex.ipynb

Wait a minute!

“staticLex in the filename? Not staticLexFull?” Yea… doesn’t look like docplex has the same capabilities as OPL, but the code sure does look nice doesn’t it. I suppose we’ll have to make these weights into constraints.

Bug fix: incorrect domain data #

I realised half way through working on adding weights that the build time decision expression is incorrect as berserkers are recruited in a different building - Hall of Order (hoo). So the total build time should be a maximum between those two times. Additionally, I’m tightening down the number of seconds in a month as there can be more than one value.

For this, I’m splitting what units from which buildings into two arrays

units = [axe_unit, lc_unit, ma_unit, serk_unit, ram_unit]
barracks_units = [axe_unit, lc_unit, ma_unit, ram_unit] # new
hoo_units = [serk_unit] #new

Splitting the build times into two decision expressions

m.barracks_build_time = m.sum([number_of_units[u.name] * u.recruit_time_in_seconds for u in barracks_units])
m.hoo_build_time = m.sum([number_of_units[u.name] * u.recruit_time_in_seconds for u in hoo_units])

In turn, split the time constraints in two as well.

ct_build_time_less_than_4_weeks_barracks = m.add_constraints([1 <= m.barracks_build_time, (m.barracks_build_time <= 4 * 7 * 24 * 3600)])
ct_build_time_less_than_4_weeks_hoo = m.add_constraints([1 <= m.hoo_build_time, (m.hoo_build_time <= 4 * 7 * 24 * 3600)])

Since there are new decision expressions, these need to come through as new KPIs

m.add_kpi(m.totalNegativeAttack, "Total negative attack strength")
m.add_kpi(m.barracks_build_time, "Barracks build time") # new
m.add_kpi(m.hoo_build_time, "Hall of Order build time") # new

Having a Python interface really does make the modeling easier. The new model is at 01-1-fix-build-times.ipynb. Here are the new results.

['totalNegativeAttack', 'hoo_build_time', 'barracks_build_time'] - strength: 935870.0, food: 20596.0
axe: 6386.0, lc: 0, ma: 0, serk: 2160.0, ram: 250.0, food: 20596.0, time: 38.04 days
 
['totalNegativeAttack', 'barracks_build_time', 'hoo_build_time'] - strength: 935870.0, food: 20596.0
axe: 6386.0, lc: 0, ma: 0, serk: 2160.0, ram: 250.0, food: 20596.0, time: 38.04 days
 
['hoo_build_time', 'totalNegativeAttack', 'barracks_build_time'] - strength: 871100.0, food: 20596.0
axe: 19340.0, lc: 0, ma: 0, serk: 1.0, ram: 250.0, food: 20596.0, time: 21.55 days
 
['hoo_build_time', 'barracks_build_time', 'totalNegativeAttack'] - strength: 870380.0, food: 20580.0
axe: 19324.0, lc: 0, ma: 0, serk: 1.0, ram: 250.0, food: 20580.0, time: 21.53 days
 
['barracks_build_time', 'totalNegativeAttack', 'hoo_build_time'] - strength: 935150.0, food: 20580.0
axe: 6370.0, lc: 0, ma: 0, serk: 2160.0, ram: 250.0, food: 20580.0, time: 38.02 days
 
['barracks_build_time', 'hoo_build_time', 'totalNegativeAttack'] - strength: 935150.0, food: 20580.0
axe: 6370.0, lc: 0, ma: 0, serk: 2160.0, ram: 250.0, food: 20580.0, time: 38.02 days

36.94 days? That’s more than a month, how did it get through? Shouldn’t this get filtered by the constraints? I’m as perplexed as you are.

Adding weights using constraints #

The current results with this model gives me a lot of axemen and can be countered easily. Adding light cavalry and mounted archer will reduce the attack strength but increase the chances of the attack coming through.

Because berserker is a special unit, the whole defending army is the counter. Therefore, I’ll be only adding three new constraints specific to just axemen, light cavalries and mounted archers axe_strength, lc_strength and ma_strength. Each will be a decision expression with the following rough pseudo code: number_of_units[u.name] * u.att_strength, where u is the unit in question. In the next portion of the post, I’ll externalise these strength weightings. For now, I’ll make them all equal to one another, i.e., weight of 1.

With the new model at 02-fixed-equal-weights.ipynb, the overall strength is about 200K lower. Remember, this is a compromise to counter the types of troops the defender has.

strength ratio: [0.33333333 0.33333333 0.33333333]

['totalNegativeAttack', 'hoo_build_time', 'barracks_build_time'] - strength: 683050.0, food: 20596.0
axe: 4814.0, lc: 1664.0, ma: 1442.0, serk: 111.0, ram: 250.0, food: 20596.0, time: 22.39 days
 
['totalNegativeAttack', 'barracks_build_time', 'hoo_build_time'] - strength: 683050.0, food: 20596.0
axe: 4814.0, lc: 1664.0, ma: 1442.0, serk: 111.0, ram: 250.0, food: 20596.0, time: 22.39 days
 
['hoo_build_time', 'totalNegativeAttack', 'barracks_build_time'] - strength: 673025.0, food: 20596.0
axe: 4985.0, lc: 1725.0, ma: 1491.0, serk: 1.0, ram: 250.0, food: 20596.0, time: 21.55 days
 
['hoo_build_time', 'barracks_build_time', 'totalNegativeAttack'] - strength: 672405.0, food: 20580.0
axe: 4979.0, lc: 1720.0, ma: 1493.0, serk: 1.0, ram: 250.0, food: 20580.0, time: 21.53 days
 
['barracks_build_time', 'totalNegativeAttack', 'hoo_build_time'] - strength: 682455.0, food: 20580.0
axe: 4807.0, lc: 1663.0, ma: 1441.0, serk: 111.0, ram: 250.0, food: 20580.0, time: 22.37 days
 
['barracks_build_time', 'hoo_build_time', 'totalNegativeAttack'] - strength: 682455.0, food: 20580.0
axe: 4807.0, lc: 1663.0, ma: 1441.0, serk: 111.0, ram: 250.0, food: 20580.0, time: 22.37 days

Shipping the model to the cloud #

At this point, I think we have a good enough base to start building a web interface to interact with the model. IBM Cloud’s Watson Studio has a Decision Optimiser environment that we can use to run the Jupyter notebook.

First, let’s condense the model to just one big cell, and externalise all modifiable variables 03-externalise.ipynb.

All the documentation seems to be pointing me to use Decision Optmizer Model Builder, which lets me prepare a model, run some DO experiments and deploy it to Watson Machine Learning. I’ve already got a model and a local environment, so instead, for this section, I’m going to deploy the model directly with Watson Machine Learning.

Prepare input and output formats #

For docplex models, IBM Cloud doc suggests using an input file and an output file. I’m not too sure at this point what take advantage of data connectors mean, but the .csv format is recommended for the input file.

Any format can be used in your Python code, but to take advantage of data connectors, use the .csv format. To read and write input and output in Python, use the commands get_input_stream("filename") and get_output_stream("filename"). See DOCPLEX API sum example.

So far so vague, what do I actually need to change? The REST API example doco finally provides some code sample in an external Github repository.

diet.py reads in set of CSVs and output a set of CSVs. More comments in the code samples below.

  • 04-1-prep-for-wml.ipynb brings the last section’s code and prepare inputs and outputs as the docs instruct
  • 04-1-optimisation-params.csv is the input CSV. get_all_inputs() reads all input CSVs. While testing locally with a local CPLEX runtime, this file lives in the same folder as the model code.
  • solution.csv is the solution CSV

Deploy the model to Watson Machine Learning service with Python #

The IBM Cloud team has created a Python library and instructions on how to deploy a DO model with Watson Machine Learning. This is great and strange at the same time. While I’m thankful for a code example, why did they create a separate Python library and not include Watson Machine Learning SDK in the official Watson SDK? Instead, their Python SDK’s documentation is hosted in a non-IBM domain.

For the sample code, the IBM team is using IBM Watson Machine Learning API v4. As the instructions are maintained by the IBM team and to protect my credentials, I won’t be posting the deployment code. Instead a test job was run and this is the results: 05-run-optimisation-job.ipynb.

Conclusion #

DOcplex not supporting staticLexFull was a bit of a bummer but there were work-arounds. With Python being a dynamic language, I found going through the docs a tad challenging since there were no examples of just the functions by themselves. The sample code were massive to navigate through.

I honestly didn’t think there would be so many hoops to jump through to deploy a DO model to Watson Machine Learning. I found it surprising that the Watson Machine Learning team maintain their own Python library seperated to the larger IBM Watson SDK teams. They also only maintain a Python library that my google-fu could find, i.e. not in any other mainstream languages.

It was clear that the IBM Cloud & Data Platform team wants me to use the Decision Optimization Model Builder (DOMB) inside Watson Studio to build a DO model and deploy it to Watson Machine Learning via Watson Studio. Maybe this is the cloud version of Cloud Pak for Data and there are certain features not available. However, if this is the same for the full enterprise version of Cloud Pak for Data, I think there will be a lot of annoyed optimisation developers who prefer to work with the desktop CPLEX studio version and like myself, a local instance of Jupyter Lab, where everything is more responsive since there is no network latency.

Once the model is deployed by uploading a ZIP file to the deployment space, I found out that I could not replace the model in Watson Studio, i.e., I could only use the REST API or the Python library to redeploy the model into another deployment space.

The consumption side of Watson Machine Learning was great. It was smooth sailing from the moment the model was deployed. For this project, I did not dive into looking at logs, metrics and traces for each job as the focus was to get my feet wet with IBM Decision Optimization. As a whole, I could do everything from start to finish, yes there were bumps along the way and hills to climb but got there eventually. I think the platform will be even better if the user journey for a desktop user is more focused.