Monday 22nd October

Last night I finished all the practical work. I managed to test a MLP with the same inputs as the KBS. I tried various certainty factors and rules with the KBS to improve performance but it could not match the MLP. At least this is a direct comparison of KBS vs NNs which all the other experiments are not.

Began write up today. I will just carry on from the draft really as there wasn't much wrong with that. Word 2007 does make it look impressive. So even if it's crap it will look good.

Sunday 20th

Rechecked scoring of KBS experiments and corrected a few mistakes. Both experiment scoring the same though PPVs are different.

A few observations about these KBS:

Is there any point in putting three rules in for the grade as by inference if it’s not one then it’s one of the other two? Would two rules be enough

There are no cases in the training set with 18+ lymph nodes but this rule does exert an influence on the other cases. Do I remove it?

Currently modifying original KBS to test this.

----------------
Now playing: The Avalanches - Frontier Psychiatrist
via FoxyTunes

Saturday 19th October

A rare weekend day with not a great deal to do. So more OU. Not much more to do now. I completed testing and scoring my flex program yesterday. I built a MLP last night and tested the same training file as I have been using to test the flex program, however I did this with the full 32 input neurons. The MLP scored very well but it will be interesting to see how it scores with the same number of inputs as the flex program. This is what I am doing now. I am modifying the KBS input files for the MLP.

I have in the mean time modified the certainty factors in the flex program to try to improve the performance but without success.

Will complete this practical work and begin writing up the results and the project this weekend.

T-25 days

----------------
Now playing: Super Furry Animals - Lazer Beam
via FoxyTunes

Friday 19th Arrrggghhhh!

Tried copying the line "prognosis is reccurence" from rule r1c to everywhere it occured in the other rules etc. Then recompiled and tried again. Rule r8c now working properly. Cannot see why this happened. See below:

C.F. : TRY : r8c
C.F. : LOOKUP : (grade is '3') = -1
C.F. : IMPLIES : cf(0.35) @ -1 -> -0.35
C.F. : LOOKUP : (prognosis is reccurence) = 0.2
C.F. : CONFIRMS : 0.2 + -0.35 -> -0.1875
C.F. : UPDATE : (prognosis is reccurence) = -0.1875
C.F. : FIRED : r8c

Looked at the certainty factor example in Chapter 3 of T396 and it doesn't simply add certainty factors together.

Start testing KBS again!

----------------
Now playing: Maximo Park - Going Missing
via FoxyTunes

Friday

Yesterday I did a number of experiments with NNs. I couldn't get a score of more than 51% for any network topography despite adjusting training cycles etc. The performance of many of the networks was remarkably similar.

I identified those cases which gave false +ve and -ve results and added them to the training sets but performance deteriorated. The data is ambiguous. Some patients have virtually the same sets of data but different outcomes. There are not enough data items to improve the performance - no ER/PR results etc.

Preprocessed the training set for the KBS. I couldn't get excel to output a .csv file so I had to produce it manually. I then copied these into word so that I could paste who strings into the console window of flex.

Today I have been running these and have noticed that rule 8c is not working properly as it is replacing the certainty from the previous rules and replaces it.

C.F. : TRY : r9c
C.F. : LOOKUP : (location is central) = -1
C.F. : LOOKUP : (grade is '1') = 1
C.F. : AND : -1 + 1 -> -1
C.F. : IMPLIES : cf(-0.5) @ -1 -> 0.5
C.F. : LOOKUP : (prognosis is reccurence) = 0.1
C.F. : CONFIRMS : 0.1 + 0.5 -> 0.55
C.F. : UPDATE : (prognosis is reccurence) = 0.55
C.F. : FIRED : r9c

C.F. : TRY : r8c
C.F. : LOOKUP : (grade is '3') = -1
C.F. : IMPLIES : cf(0.35) @ -1 -> -0.35
C.F. : UPDATE : (prognosis is recurrence) = -0.35
C.F. : FIRED : r8c

C.F. : TRY : r7c
C.F. : LOOKUP : (grade is '2') = -1
C.F. : IMPLIES : cf(0.1) @ -1 -> -0.1
C.F. : LOOKUP : (prognosis is recurrence) = -0.35
C.F. : CONFIRMS : -0.35 + -0.1 -> -0.415
C.F. : UPDATE : (prognosis is recurrence) = -0.415
C.F. : FIRED : r7c

I need to sort this before continuing.

Also if the CF.:CONFIRMS lines are supposed to be adding the numbers together they are wrong.

C.F. : TRY : r11c
C.F. : LOOKUP : (size is less_than_15) = 1
C.F. : IMPLIES : cf(-0.25) @ 1 -> -0.25
C.F. : LOOKUP : (prognosis is reccurence) = -0.5
C.F. : CONFIRMS : -0.5 + -0.25 -> -0.625 [-0.5+ -0.25 = -0.75]
C.F. : UPDATE : (prognosis is reccurence) = -0.625
C.F. : FIRED : r11c

Wednesday 17th October 11p.m.

A day of testing NNs. All the MLPs I tested regardless of the number of training iterations or hidden layer neurons gave very similar results. So I analysed the results of Exp E, as it was fairly representative, for false +ves and -ves. I then produced a new training file of some of these and the original random cases but still with 50 cases.

The results of an MLP with 10 hidden layer neurons and 50000 training iterations were poorer than using the random set.

Next try a training set of all false +ves and -ves plus orginal training set.

Wednesday 17th October 11a.m.

More experimenting. Three days to complete the experiments and then a couple of weeks to write up. Well that's the plan anyway.

First up complete neural network experiments. Not happy with the ones I've carried out before they are to ad hoc. I will organise them better this time.

So first create an MLP and work out the optimum number of training cycles. Do each experiment at least twice because NNs do not always cluster data the same way each time they are trained.

Score with score tool and calculate sensitvity, specificity and PPV.

Then experiment using different numbers of hidden layer neurons to find optimum.
Then, well I'll see how it's going.

----------------
Now playing: Jeff Wayne - Dead London
via FoxyTunes